Improving Conversations: Lessons Learnt from Manual Analysis of Chatbot Dialogues

Abstract

Analysing and improving chatbot dialogues – so-called chatbot ‘training’ – is key to the successful implementation and maintenance of chatbots for customer service. Nevertheless, the details of this practice and what service providers may learn from the analysis of such dialogues is not investigated in current research on chatbots. As a first step towards bridging this gap in existing knowledge, we present a study of the qualitative analysis of chatbots dialogues in the context of the customer service department of a large telecom provider. In total 406 dialogues, randomly sampled from all chatbot dialogues during a four-week period, were included in the analysis. The analysis concerned the chatbot’s ability to resolve customers’ requests, the quality in the chatbot dialogues, and suggestions for improvements of the chatbot knowledge base generated through the analysis. The findings shed light on characteristics of successful and unsuccessful chatbot dialogues and the kind of improvements that may be derived from such analysis. On the basis of the findings we summarize implications for theory and practice, and suggest future research.