UX of Chatbots: An Exploratory Study on Acceptance of User Experience Evaluation Methods

Abstract

Companies increasingly invest in designing, developing, and evaluating conversational agents, mainly text-based chatbots. Chatbots have become the main and the fastest communication channel for providing customer service and helping users interact with systems and, consequently, obtain the requested information. Despite this, the potential market for chatbots,there is still too little known about evaluating the quality of chatbots from a User eXperience (UX) perspective, i.e., the emotions, feelings, and expectations of users towards this application. Besides, relatively little research addresses the feasibility and applicability of UX methods (generic or not) or how to adapt them (if necessary) to evaluate chatbots. The goal of this research is to investigate the adequacy, feasibility of use, and acceptance by users of UX methods when employed to evaluate a chatbot. To achieve this objective, we conducted an exploratory study comparing three UX methods: AttrakDiff, Think Aloud, and Method for the Assessment of eXperience. We compared these methods by assessing the degree of ease of use, usefulness, self-predicted future use from end-users. Also, we performed follow-up interviews to understand users’ perceptions about each method. The results show that users preferred to use the Think Aloud method due to the ease and freedom that the user has to express their positive and negative emotions/feelings while using the chatbot. Based on the results, we believe that combining the three methods was essential to capture the whole user experience when using the chatbot.