Empirical Assessment of User Satisfaction with American and Chinese AI Chatbots

Abstract

As artificially intelligent chatbots grow in popularity, exploring user satisfaction has become an important topic. This study deviates from the conventional questionnaire approach by adopting a balanced paired design. We employed 40 users to participate in usability testing to assess their satisfaction with the responses of two prominent chatbots (ChatGPT 4.0 from the US and inChat from China) over five domains (daily life, the workplace, advertising copy, current affairs commentary, and translation). We conducted a comparative analysis based on three demographic variables: gender, experience with chatbots, and generation (i.e., age). Empirical results revealed that the participants were more satisfied with inChat's responses overall and for daily life scenarios in particular than with those of ChatGPT 4.0. Further, female participants expressed higher satisfaction with inChat's commentary on current affairs than did male participants. Prior use of chatbots did not significantly influence satisfaction levels; however, Baby Boomers (i.e., those born between 1946 and 1964) showed a notably higher appreciation for inChat's translation capabilities and higher satisfaction overall compared to Generation Y (i.e., those born between 1981 and 1996).