An introduction to the statistical evaluation of fluency measures with signal detection theory

Abstract

Fluency represents the learned ability to respond quickly, effortlessly, and accurately to a given stimuli. The fluent application of a skill, however, requires frequent and deliberate practice on all relevant subskills, not simply the repetition of subskills that is already fluent. Dancers, for example, learn best through marking, where they practice only partial movements of a performance. Diagnosing the source of the disfluency is critical for educators. Judgments grounded on data, statistical models, and even informal prediction models, however, outperform those based on intuition alone. Teachers can easily and accurately select the students in most need of supplemental instructions or support through the use of diagnostic or classification systems. This chapter describes the basic methods recommended for the development and evaluation of classification systems using a framework called signal detection theory. We present the theoretical basis for signal detection, and methods for statistically evaluating diagnostic decisions in education, which seek to balance time, clarity, and accuracy. The methods can be applied to any screener or test, continuous or ordinal, including many measures that are available in education, used to gauge the likely accomplishment of a relevant criterion. To illustrate the methods, we select Dynamic Indicators of Basic Early Literacy Skills (DIBELS; 6th Ed.) measures as an example of screening system and use the Stanford Achievement Test (10th Ed.) as the criterion measure.