A live-user evaluation of incremental dynamic critiquing

Abstract

Feature critiquing has emerged as an important feedback strategy for conversational recommender systems as it offers a useful balance between user effort and recommendation efficiency. Dynamic critiquing has recently been presented as an extension to conventional (single-feature) critiquing that supports the simultaneous critiquing of multiple features. To date, dynamic critiquing has been evaluated through a variety of artificial user trials to demonstrate its potential advantages, when it comes to improving recommendation efficiency and quality. However these advantages have never been verified through any large-scale user trial. The contribution of this paper is that we present the results of such an evaluation, which confirms the advantages of dynamic critiquing in a realistic online, e-commerce setting. Furthermore we investigate the impact of implicitly maintaining session specific user models to influence the selection of compound critiques. These models are incrementally constructed as the user critiques example recommendations from cycle to cycle. Our live-user evaluation also enabled us to analyse how real users interact with the compound critiques that are produced in this way. The results demonstrate that our incremental critiquing approach has the capability of generating more relevant critique options, and that users frequently recognise the benefits associated with using these as feedback options, leading to significantly shorter recommendation sessions. © Springer-Verlag Berlin Heidelberg 2005.