Generalizability Theory: An Analysis of Variance Approach to Measurement Problems in Educational Assessment

Abstract

Of increasing interest to the educational assessment researchers is the reliability of scores obtained by various measurement procedures like tests, rating scales, surveys, and observation forms. Traditional methods of reliability based on Classical Test Theory (CTT) consider only one source of measurement errors. Generalizability Theory (GT) extends CTT by providing a flexible and practical framework for estimating the effects of multiple sources of measurement errors through an application of analysis of variance procedures. It helps educational assessment researchers to determine appropriate conditions in terms of items, occasions, raters conducive to obtaining an optimal level of score reliability. This paper highlights the utility and applicability of the GT analysis in the educational assessment research. The paper begins with a brief history of the GT tracing its growth and development and then discusses its defining features and advantages and disadvantages. The paper illustrates the application of the GT in the educational assessment research. The paper closes with a recommendation for educational assessment researchers to take advantage of the GT in the development of measurement procedures and further research agenda in this area.