Making our “meta-hypotheses” clear: heterogeneity and the role of direct replications in science

Abstract

This paper argues that some of the discussion around meta-scientific issues can be viewed as an argument over different “meta-hypotheses” – assumptions made about how different hypotheses in a scientific literature relate to each other. I argue that, currently, such meta-hypotheses are typically left unstated except in methodological papers and that the consequence of this practice is that it is hard to determine what can be learned from a direct replication study. I argue in favor of a procedure dubbed the “limited homogeneity assumption” – assuming very little heterogeneity of effect sizes when a literature is initiated but switching to an assumption of heterogeneity once an initial finding has been successfully replicated in a direct replication study. Until that has happened, we do not allow the literature to proceed to a mature stage. This procedure will elevate the scientific status of direct replication studies in science. Following this procedure, a well-designed direct replication study is a means of falsifying an overall claim in an early phase of a literature and thus sets up a hurdle against the canonization of false facts in the behavioral sciences.