User Experience of Alexa, Siri and Google Assistant When Controlling Music – Comparison of Four Questionnaires

Abstract

We evaluate user experience (UX) when users play and control music with three smart speakers: Amazon’s Alexa Echo, Google Home and Apple’s Siri on a HomePod. For measuring UX we use four established UX metrics (AttrakDiff, SASSI, SUISQ-R, SUS). We investigated the sensitivity of these four questionnaires in two ways: firstly, we compared the UX reported for each of the speakers, secondly, we compared the UX of completing easy single tasks and more difficult multi-turn tasks with these speakers. We find that the investigated questionnaires are sufficiently sensitive to show significant differences in UX for these easy and difficult tasks. In addition, we find some significant UX differences between the tested speakers. Specifically, all tested questionnaires, except the SUS, show a significant difference in UX between Siri and Alexa, with Siri being perceived as more user friendly for controlling music. We discuss implications of our work for researchers and practitioners.