Appropriately estimating the standardized average treatment effect with missing data: A simulation and primer

Abstract

Reporting standardized effects in randomized treatment studies aids interpretation and facilitates future meta-analyses and policy considerations. However, when outcome data are missing, achieving an unbiased, accurate estimate of the standardized average treatment effect, sATE, can pose challenges even for those with general knowledge of missing data handling, given that the sATE is a ratio of a mean difference to a (within-group) standard deviation. Under both homogeneity and heterogeneity of variance, a Monte Carlo simulation study was conducted to compare missing data handling strategies in terms of bias and accuracy in the sATE, under specific missingness patterns plausible for randomized pretest posttest studies. Within two broad missing data handling approaches, maximum likelihood and multiple imputation, modeling choices were thoroughly investigated including the analysis model, variance estimator, imputation algorithm, and method of pooling results across imputed datasets. Results demonstrated that although the sATE can be estimated with little bias using either maximum likelihood or multiple imputation, particular attention should be paid to the model and variance estimator, especially at smaller sample sizes (i.e., N = 50). Differences in accuracy were driven by differences in bias. To improve estimation of the sATE in practice, recommendations and a software demonstration are provided. Moreover, a pedagogical explanation of the causes of bias, described separately for the numerator and denominator of the sATE is provided, demonstrating visually how and why bias occurs with certain methods.