Classical test theory

Abstract

Classical test theory (CTT) was first developed and predominated in measurement circles during the early to middle twentieth century and is “a measurement theory which consists of a set of assumptions about the relationships between actual or observed test scores and the factors that affect these scores, which are generally referred to as error” (Association of Language Testers in Europe, 1998: 138). “At the heart of CTT is the assertion that an observed score is determined by the actual state of the unobservable variable of interest plus error contributed by all other influences on the observable variable. The actual state of the unobserved variable is its hypothetical true score” (DeVellis, 2006: S50). Clearly, the notions of error and true scores are central to CTT and therefore to all else that follows in this chapter. The recognition that all measurement contains errors can be traced back at least as far as Spearman (1904: 76), who observed that individuals created “accidental deviations” in any measurement which become “variable errors” for groups of individuals. Put in the parlance of CTT, observed scores (i.e., the examinees’ actual scores on a test) contain errors (unsystematic effects due to factors not being measured), which in turn contribute to error variance (unsystematic variation in the scores that is due solely to random errors). Such error variance is taken to be random because it can arise from a variety of extraneous, non-systematic sources, which typically have nothing to do with the purposes for which the test was designed, for example, sources of error arise in the environment (e.g., noise, heat, lighting, etc.), administration procedures (e.g., directions, equipment, timing, etc.), scoring procedures (subjectivity, math errors, scorer biases, etc.), test items (e.g., item types, item quality, test security, etc.), the examinees themselves (e.g., health, fatigue, motivation, etc.), and so forth (Brown, 2005: 171-75). Unless the scores on a particular test are completely random (i.e., 100% error), some pro-portion of the observed score variance that results from the test will be attributable to the construct that the test was designed to measure. True scores are hypothetical representations of the scores that would result if there were no errors in measurement. Variation among examinees in their hypothetical true scores is called true score variance. Logically, any set of observed scores will be made up of both true score variance and error variance. Let’s represent these relationships as follows: Total test variance = true score variance + error variance VarTotal 1/4 VarTrue Score p VarError o1p This notion that observed score variance is made up of true score variance plus error variance underlies the entire framework of CTT.