When there is no progress with a task-oriented chatbot: A conversation analysis

Abstract

Task-oriented chatbots are increasingly prevalent in our daily life. Research effort has been devoted to advancing our understanding of users' interaction with conversational agents, including conversation breakdowns. However, most research attempts were limited to obversions from a relatively short duration of user interaction with chatbots, where users were aware of being studied. In this study, we conducted a conversation analysis on a three-month conversation log of users conversing with a chatbot of a banking institution. The log consisted of 1,837 users' conversations with this chatbot with 19,449 message exchanges. From this analysis, we show that users more often failed to make a progress in a conversation when they requested information than when they provided information. Furthermore, we uncovered five kinds of intention gaps unexpected to the chatbot, and five major behaviors users adopted to cope with non-progress.