Cavalier use of inferential statistics is a major source of false and irreproducible scientific findings

Abstract

I uncover previously underappreciated systematic sources of false and irreproducible results in natural, biomedical and social sciences that are rooted in statistical methodology. They include the inevitably occurring deviations from basic assumptions behind statistical analyses and the use of various approximations. I show through a number of examples that (a) arbitrarily small deviations from distributional homogeneity can lead to arbitrarily large deviations in the outcomes of statistical analyses; (b) samples of random size may violate the Law of Large Numbers and thus are generally unsuitable for conventional statistical inference; (c) the same is true, in particular, when random sample size and observations are stochastically dependent; and (d) the use of the Gaussian approximation based on the Central Limit Theorem has dramatic implications for p‐values and statistical significance essentially making pursuit of small significance levels and p‐values for a fixed sample size meaningless. The latter is proven rigorously in the case of one‐sided Z test. This article could serve as a cautionary guidance to scientists and practitioners employing statistical methods in their work.