The puzzling relationship between multi-laboratory replications and meta-Analyses of the published literature

Abstract

What is the best way to estimate the size of important effects? Should we aggregate across disparate findings using statistical meta-Analysis, or instead run large, multi-laboratory replications (MLR)? A recent paper by Kvarven, Stromland and Johannesson (Kvarven et al. 2020 Nat. Hum. Behav. 4, 423-434. (doi:10.1038/s41562-019-0787-z)) compared effect size estimates derived from these two different methods for 15 different psychological phenomena. The authors reported that, for the same phenomenon, the meta-Analytic estimate tended to be about three times larger than the MLR estimate. These results are a specific example of a broader question: What is the relationship between meta-Analysis and MLR estimates? Kvarven et al. suggested that their results undermine the value of meta-Analysis. By contrast, we argue that both meta-Analysis and MLR are informative, and that the discrepancy between the two estimates that they observed is in fact still largely unexplained. Informed by re-Analyses of Kvarven et al. s data and by other empirical evidence, we discuss possible sources of this discrepancy and argue that understanding the relationship between estimates obtained from these two methods is an important puzzle for future meta-scientific research.