Testing Chatbots Using Metamorphic Relations

Abstract

Modern-day demands for services often require an availability on a 24/7 basis as well as online accessibility around the globe. For this sake, personalized software systems, called chatbots, are applied. Chatbots offer services, goods or information in natural language. These programs respond to the user in real-time and offer an intuitive and simple interface to interact with. Advantages like these makes them increasingly popular. Chatbots can even act as substitutes for humans for specific purposes. Since the chatbot market is growing, chatbots might outperform and replace classical web applications in the future. For this reason, ensuring correct functionality of chatbots is of high and increasing importance. However, since different implementations and user behavior result in unpredictable results, the chatbot’s output is difficult to predict and classify as well. In fact, testing of chatbots represents a challenge because of the unavailability of a test oracle. In this paper, we introduce a metamorphic testing approach for chatbots. In general, metamorphic testing can be applied to situations where no expected values are available. In addition, we discuss how to obtain test cases for chatbots, i.e. sequences of interactions with a chatbot, in an according manner. We demonstrate our approach using a hotel booking system and discuss first experimental results.