Variability of Statistics in Generalizability Theory

Abstract

Estimates of variance components, error variances , generalizability coefficients , and so on, like all statistics, are subject to sampling variability. Traditionally, such variability is quantified through estimated standard errors and/or confidence intervals. Cronbach et al. (1972) recognized the importance of this topic for generalizability analyses and gave it more than passing attention, although at that time statistical methodologies for addressing the topic were limited. Subsequently, in the generalizability theory literature, Smith (1978, 1982) considered standard errors of estimated variance components, Brennan (1992a) summarized some procedures for establishing standard errors and confidence intervals, Brennan et al. (1987) considered bootstrap and jackknife procedures, Betebenner (1998) examined a relatively new procedure for establishing confidence intervals , Wiley (2000) studied the bootstrap, and Gao and Brennan (2001) provided examples from th e performance assessment literature. In the statistical literature, assuming score effects are normally distributed , Searle et al. (1992) treat in detail standard errors for estimated variance components,' and Burdick and Graybill (1992) provide a very comprehensive and readable treatment of confidence intervals for variance components and various ratios of them. This chapter summarizes most methodologies that have been developed to estimate standard errors and confidence intervals for (estimated) vari-1 In many resp ects, Searle et a l. (1992) build on the seminal work reported two decades earlier by Searle (1971) .