Are all perfect automation schemas equal? testing differential item functioning in programmers versus the general public

Abstract

Humans routinely overestimate the reliability of automated systems. Given people’s different experiences in using automated systems, their schemas regarding the performance of those systems may be more or less developed. Extant research has assessed people’s automation schema using the Perfect Automation Schema (PAS) scale, a self-report scale comprising two facets (i.e., high expectations and all-or-none thinking). We used item response theory to determine the extent to which two populations (the public versus computer programmers) responded to self-report items similarly. Computer programmers (n = 245) and members of the public (n = 285) completed the PAS scale in the laboratory and Amazon’s Mechanical Turk, respectively. Results showed that items of the high expectations facet functioned differently across the two subgroups. In addition, we observed that the high expectations facet as a whole functioned differently across the two groups. Theoretical and practical implications are discussed.