Adjustment of p-values for multiple hypotheses

Abstract

It is quite common to investigate multiple hypotheses in a single study, which increases the probability of Type I errors. This can be dealt with in various ways. A researcher may have various reasons for testing multiple hypotheses in the same study, for example to investigate the effect on several outcome variables, compare more than two groups or undertake separate analyses for subgroups. Different adjustment methods Consider a study where six hypothesis tests are performed. If all tests are made at a significance level of 5 %, each of them will have a 5 % probability of making a Type I error, that is, erroneously rejecting the null hypothesis (1). The probability of a Type I error in at least one of the hypothesis tests, also referred to as the family-wise error rate (FWER) (2), will then be substantially higher than 5 %, and at worst almost 30 %. Sometimes it is desirable to control this error rate to prevent it from exceeding a pre-defined threshold, for example a significance level of 5 %. The simplest method is a so-called Bonferroni correction. This means multiplying the p-values by the number of hypotheses, in this case six, before comparing with the significance level. However, the Bonferroni correction is very conservative, which means that the statistical power, and thereby the Adjustment of p-values for multiple hypotheses | Tidsskrift for Den norske legeforening STIAN LYDER SEN