Response Time Data as Validity Evidence: Has It Lived Up To Its Promise and, If Not, What Would It Take to Do So

Abstract

The widespread use of computers in test delivery provides easy access to data on response processes; it is possible to track and trace events like clicks of the mouse, movement of text or objects, or the time that test takers use to respond to items (i.e., response time). Response time (RT) is typically defined as the time a test taker uses to complete an item or task, beginning from the initial presentation of the task to the time at which the complete response is logged. RT has attracted substantial attention in recent years as it offers a promising window into test takers’ cognitive processes and thus the construct being measured (Huff & Sireci, 2001; Schnipke & Scrams, 1997). Molenaar (2015) argues that “the natural variability in response times can give valuable information for psychological and educational inferences about response processes and solution strategies” (p. 177). As such, RT offers an opportunity to build validity evidence for a test. This expectation that RT data can and should be a source of validity evidence originates in part from its association with response processes; the time that a test taker takes to process and respond to an item is a natural corollary of the psychological, cognitive or thinking processes activated during the act of responding to an item. Indeed, the Standards for Educational and Psychological Testing (AERA,