What does the system usability scale (SUS) measure?: Validation using think aloud verbalization and behavioral metrics

Abstract

The System Usability Scale (SUS) is widely used as a quick method for measuring usability; however, past research showed there is only a weak relationship between SUS scores and one behavioral usability measure, and alternatively, SUS corresponds more strongly with user preference. This suggests that the underlying constructs of the SUS may not be well understood. In this study, participants were asked to think aloud while completing a usability test and filling out the SUS. Correlations showed no relationship between behavioral performance and SUS scores. Instead, a relationship was observed between SUS scores and perceived success. Furthermore, participants described a variety of reasons for selecting their SUS responses that were unrelated to the usability of the system, which we have termed rationalizations. This suggests that the SUS is constructed of a combination of experiential components, including attitudinal perceptions. Consequently, SUS scores may be more helpful as a tool for comparison (between competitors, iterations, etc.,) or when used in conjunction with formative usability testing methods to provide a holistic view of real and perceived user experience.