The Role of p-Values in Judging the Strength of Evidence and Realistic Replication Expectations

Abstract

p-Values are viewed by many as the root cause of the so-called replication crisis, which is characterized by the prevalence of positive scientific findings that are contradicted in subsequent studies. The spectrum of proposed solutions includes redefining statistical significance, abandoning the concept of statistical significance, or eliminating the use of p-values altogether. The unintended consequence of these proposals has been confusion within the scientific community, especially in the absence of consensus or clear alternatives. The goal of this article is to reframe the perceived replication crisis. I argue that this crisis is to a large extent the result of excessive optimism based on unknowingly (and sometimes knowingly) overstated evidence. As a remedy, I suggest a four-part guide to navigating statistical inference with p-values that is accessible for scientists. Examples taken from pharmaceutical drug development for heart failure illustrate key concepts.