Modelling goal modifications in user simulation

Abstract

User simulation is frequently used to evaluate spoken dialogue systems. Previous work in this field primarily focused on the users’ interaction behaviour. Less attention has been paid to the users’ goals, how they relate to the system capabilities, and how they may change over the course of a dialogue. User goals may be underspecified or over-specified in comparison to the attributes the system uses to describe objects in its domain. Goal modifications can occur, e.g., if no database entry matches the user query. Analysing empirical data, we show that the definition of possible goals and goal modifications impacts the results of a user test significantly in terms of system performance and discovered usability problems. We propose a task modelling approach able to represent such variable goals in user simulations. Dialogues simulated using this approach are shown to be more similar to empirical data than dialogues simulated with conventional task models.