How to statistically show the absence of an effect

Abstract

In experimental studies, the lack of statistical significance is often interpreted as the absence of an effect. Unfortunately, such a conclusion is often a serious misinterpretation. Indeed, non-significant results are just as often the consequence of an insufficient statistical power. In order to conclude beyond reasonable doubt that there is no meaningful effect at the population level, it is necessary to use proper statistical techniques. The present article reviews three different approaches that can be used to show the absence of a meaningful effect, namely the statistical power test, the equivalence test, and the confidence interval approach. These three techniques are presented with easy to understand examples and equations are given for the case of the two-sample t-test, the paired-sample t-test, the linear regression coefficient and the correlation coefficient. Despite the popularity of the power test, we recommend using preferably the equivalence test or the confidence interval.