Reversing Bonferroni

Abstract

It is common for conclusions of empirical studies to depend on multiple significant outcomes. This practice may seem reasonable, but it has some unintended effects. In particular, the compound Type I error rate for multiple studies (the likelihood of concluding that an effect exists when it does not) can be much lower than that of the individual studies. This in itself is not a problem since a low Type I error rate is desirable. However, there is also an accompanying drop in power, meaning that the probability of finding support for a true effect is low. Currently, there is no standard statistical method for dealing with the hyper-conservative error rate and accompanying low power that results from investigations requiring multiple significant outcomes. Here, we propose a novel solution to this problem: We show that it is sometimes appropriate to reverse the logic of the classic Bonferroni correction and increase the significance criterion in order to maintain an intended compound Type I error rate across multiple tests. This reverse Bonferroni approach dramatically improves statistical power and encourages careful planning of statistical analyses prior to data collection. To avoid adding to the list of questionable research practices that seem to contaminate some psychological research, we suggest that reverse Bonferroni be restricted to situations where authors pre-register their analysis plans.