Inferential misconceptions and replication crisis

Abstract

Misinterpretations of the p value and the introduction of bias through arbitrary analytical choices have been discussed in the literature for decades. Nonetheless, they seem to have persisted in empirical research, and criticisms of p value misuses have increased in the recent past due to the non-replicability of many studies. Unfortunately, the critical concerns that have been raised in the literature are scattered over many disciplines, often linguistically confusing, and differing in their main reasons for criticisms. Misuses and misinterpretations of the p value are currently intensely discussed under the label “replication crisis” in many academic disciplines and journals, ranging from specialised scientific journals to Nature and Science. In a drastic response to the crisis, the editors of the journal Basic and Applied Social Psychology even decided to ban the use of p values from future publications at the beginning of 2015, a fact that has added fuel to the discussions in the relevant scientific forums. Finally, in March 2016, the American Statistical Association released a brief statement on p values that explicitly addressed misuses and misinterpretations. In this context, we systematise the most serious flaws related to the p value and discuss suggestions of how to prevent mistakes and reduce the rate of false discoveries in the future.