Invited commentary: Evidence-based evaluation of p values and bayes factors

Abstract

Despite clear deficiencies of the p value as a summary of statistical evidence, compelling alternatives with strong theoretical justification, such as the Bayes factor and the related likelihood ratio, are rarely presented in epidemiologic publications. Comparison of the historical performance of the p value with that of its competitors in the epidemiologic literature may help epidemiologists evaluate whether Bayes factors or likelihood ratios lead to conclusions more quickly and reliably than a p value, given the same data. Empirical evidence presented by Ioannidis (Am J Epidemiol 2008;168:374-83) demonstrates that findings with p values near 0.05 tend not to be confirmed in future studies. Similarly, Bayes factors interpret p values near 0.05 as having, at best, promising evidence against the null hypothesis. However, the different types of Bayes factors require empirical evaluation of their performance in practice. P values remain popular because miniscule p values are unlikely to mislead and p values do not require alternative hypotheses. Publishing p values near 0.05 could be considered a low-threshold screen to allow many (possibly null) results to be published for follow-up consideration. Meta-analyses and studies meant to decisively convince skeptics require a stronger standard (p values much below 0.05) and a Bayes factor to interpret the p value and to facilitate incorporation of background expertise necessary for drawing comprehensive conclusions.