The frequentist facade and the flight from evidential inference

Abstract

Recent critics of the use of significance tests (e.g. Cohen, 1994) recommend the use of confidence intervals. They use Bayes' theorem in frequentist examples aimed at illustrating the 'odds-against-chance fantasy' (Carver, 1978). It is argued below that their criticisms are inappropriate and confused. In particular, they have confounded competing definitions of probability as relative frequency and as a measure of evidence in the single case. Moreover, they fail to address the very same issue (that they have just raised in the context of significance testing) in their recommended or preferred context of confidence intervals. This is demonstrated with an analogous Bayesian example of the evidentiary interpretation of confidence intervals. As these different definitions of probability are definitions, the paper concludes with the advice that each one of us must decide where we stand on the issue of probabilistic evidence in a single research application, as we do with other criteria for evaluating the scientific implications of a single piece of research.