Advanced Social Interaction with Agents

Abstract

In this paper, we apply a dialog evaluation Interaction Quality (IQ) frame- work to human-computer customer service dialogs. IQframework can be used to pre- dict user satisfaction at an utterance level in a dialog. Such a rating framework is use- ful for online adaptation of dialog system behavior and increasing user engagement through personalization. We annotated a dataset of 120 human-computer dialogs from two customer service application domains with IQ scores. Our inter-annotator agreement (ρ = 0.72/0.66) is similar to the agreement observed on the IQ annota- tions of publicly available bus information corpus. The IQ prediction performance of an in-domain SVM model trained on a small set of call center domain dialogs achieves a correlation of ρ = 0.53/0.56 measured against the annotated IQ scores. A generic model built exclusively on public LEGO data achieves 94%/65% of the in- domain model’s performance. An adapted model built by extending a public dataset with a small set of dialogs in a target domain achieves 102%/81% of the in-domain model’s performance.