Creating falseness—How to establish statistical evidence of the untrue

Abstract

Null hypothesis significance testing is the typical statistical approach in search of the truthfulness of hypotheses. This method does not formally consider the prior credence in the hypothesis, which affects the chances of reaching correct conclusions. When scientifically implausible or empirically weakly supported hypotheses are tested, there is an increased risk that a positive finding in a test in fact is false positive. This article argues that when scientifically weakly supported hypotheses are tested repeatedly—such as when studying the clinical effects of homeopathy—the accumulation of false positive study findings will risk providing false evidence also in systematic reviews and meta-analyses. False positive findings are detrimental to science and society, as once published, they accumulate persistent untrue evidence, which risks giving rise to nonpurposive research programmes, policy changes, and promotion of ineffective treatments. The problems with false positive findings are discussed, and advice is given on how to minimize the problem. The standard of evidence of a hypothesis should depend not only on the results of statistical analyses but also on its a priori support. Positive findings from studies investigating hypotheses with poor theoretical and empirical foundations should be viewed as tentative until the results are replicated and/or the hypothesis gains more empirical evidence supporting it as likely to be true.