Comparative judgment

Abstract

In the early 20th Century, Thurstone introduced the Law of Comparative Judgment as a scaling model for the method of paired comparisons, but it is only in the past 30 years that it has re-emerged as a mainstream approach in educational assessment. This re-emergence has particularly taken place in contexts where complex constructs are assessed and where expert judgment and holistic evaluation are valued, including examination standard maintenance, and the assessment of writing, design, and the creative arts. Technological advancements have furthered this revival by streamlining the otherwise laborious method through the distribution of subsets of pairs across judges and using selective algorithms to prioritize the most informative pairwise judgments. An extensive review of the literature has shown Comparative Judgment to be a highly reliable assessment method for a wide range of constructs. Moreover, it has been shown in certain contexts to have good validity evidence in terms of concurrent and criterion validity, as well as being highly sensitive to construct-relevant variance, although further research is required to validate the method across a broader range of constructs. Given the proliferation of research into Comparative Judgment in just the past decade, the next 30 years should prove to be even more fruitful in terms of the scope of its educational applications and further enhancement of its reliability and validity across these applications.