Using Extant Data to Improve Estimation of the Standardized Mean Difference

Abstract

This article presents methods for using extant data to improve the properties of estimators of the standardized mean difference (SMD) effect size. Because samples recruited into education research studies are often more homogeneous than the populations of policy interest, the variation in educational outcomes can be smaller in these samples than is reflective of the true variation in the population. This affects effect size estimation since the sample standard deviation is used in the denominator of the SMD. We propose leveraging extant data on sample variance estimates from multiple studies, made available via clearinghouse databases such as the What Works Clearinghouse, to standardize a mean difference. This allows effect sizes to be benchmarked across a common and broad population, thus enabling better comparability across studies and interventions. We derive the new estimators of the population variance and the corresponding SMD, which pool sample variances from multiple studies using both an analysis of variance and a meta-analytic framework. We demonstrate the properties of these estimators via analytic and simulation results and offer recommendations for when these estimators are appropriate in practice.