A more principled use of the p-value? Not so fast: A critique of Colquhoun's argument

Abstract

The usefulness of the statistic known as the p-value, as a means of quantifying the strength of evidence for the presence of an effect from empirical data has long been questioned in the statistical community. In recent years, there has been a notable increase in the awareness of both fundamental and practical limitations of the statistic within the target research fields and especially biomedicine. In this article, I analyse the recently published article (Colquhoun 2017 R. Soc. open sci. 4, 171085 (doi:10.1098/ rsos.171085)) which, in summary, argues that with a better understanding and thus more appropriate use of the statistic, many of the aforementioned limitations can be addressed. In particular, I demonstrate that the (often implicit) premises of this counterargument are questionable, in some cases arguably inconsistent, and that therefore the counterargument provides little if any justification for the continued use of the p-value. Additionally, my analysis should help researchers seeking to interpret their empirical data by illustrating the nuanced nature and the multiplicity of statistical, methodological and epistemological issues which must be considered in this process.