Towards Validating a Chatbot Usability Scale

Abstract

A chatbot usability questionnaire (CUQ) was designed to measure the usability of chatbots. Study objectives: 1) to test the construct validity of CUQ (i.e. does it differentiate between chatbots that we rank as having poor, average or good usability), 2) to assess the intra-rater reliability of CUQ (i.e. do participants provide the same answers/scores when assessing the usability of the same chatbots two weeks apart), and 3) to undertake exploratory factor analysis to study the underlying factors that CUQ measures. Three chatbots were selected by co-authors that were regarded as having good, average and poor usability. Participants used each of the chatbots and completed the CUQ scale for each. Participants repeated this process two weeks later to facilitate the measurement intra-rater variability. Paired t-tests were used to compare CUQ scores from each of the three chatbots. Exploratory factor analysis was used to identify the factors within the CUQ. Paired t-tests and correlation was used to measure intra-rater reliability. There was a total of 156 CUQ survey completions (26 participants completed the CUQ for 3 different chatbots and for 2 rounds: 26 * 3 * 2 = 156). Intra-rater reliability was supported as there was a good correlation between how participants completed the CUQ for the same chatbot at approximately two weeks apart (r > 0.7). As a form of construct validity, the CUQ scores for each of the three chatbots were statistically significant (p < 0.05). Factor analysis shows that the CUQ measures four factors 1) personality, 2) user experience, 3) error handling and 4) onboarding of the chatbot.