Performance evaluation of text-oriented artificial chat operation system (TACOS)

Abstract

Many countries are facing aging societies where loneliness and decreased number of communication seriously limit older adults from maintaining a decent quality of life. To tackle this problem, we have designed a chat-bot system to provide more communication chances for older adults, especially those who have fewer opportunities to converse with actual people (e.g., those living alone). With our system, the user converses about a story or topic presented in a photo in a manner and style that are familiar to the user. This approach is based on the original “coimagination method” protocol, which is a group conversation method designed to maintain a balance between listening and speaking with strict rules. Previously, we conducted a preliminary user study with several participants and the prototype system to confirm its working processes, and the initial results indicated that the protocol seemed to work well. However, we did not integrate nor evaluate our system using performance indicators when we conducted the experiment (e.g., how many users can connect to the system at once, etc.). Therefore, in this paper, we explore a dialog system called TACOS (Text-oriented Artificial Chat Operation System), which is an intermediate program that communicates with many applications and a variety of dialog systems. We measure the performance of the system by simulating three essential aspects: the number of virtual users, the frequency of the question data, and the total experiment time. The simulated results show that currently, the system can process the requests of 40 users, where each user asks a question to the system every 5 s during a 5 min period. Our results indicate that the proposed system can effectively process the amount of requests that will be tested in future experiments. In terms of data storage, the proposed system can handle datum quantities from ten thousand to one hundred thousand, and there seems to be no critical performance delay when the emulated client interacts with TACOS.