Holm’s Sequential Bonferroni Procedure

Abstract

The more statistical tests we perform, the more likely we are to reject the null hypothesis when it is true (i.e., a "false alarm," also called a "Type 1" error). This is a consequence of the logic of hypothesis testing: We reject the null hypothesis for rare events, and the larger the number of tests, the easier it is to find rare events which are false alarms. This problem is called the inflation of the alpha level. In order to be protected from it, one strategy is to correct the alpha level when performing multiple tests. Making the alpha level more stringent (i.e., smaller) will create less errors, but it may also make it harder to detect real effects. The most well known correction is called the Bonferroni correction, it consists in multiplying each probability by the total number of tests performed. A more powerful (i.e., more likely to detect an effect it it exists) sequential version has been proposed by Holm in 1979. In Holm's sequential version, the tests need first to be performed in order to obtain their "p-values." The tests are then ordered from the one with the smallest p-value to the