Sensitive and Sturdy $p$-Values

Abstract

We introduce new criteria for evaluating test statistics based on the $p$-values of the statistics. Given a set of test statistics, a good statistic is one which is robust in being reasonably sensitive to all departures from the null implied by that set. We present a constructive approach to finding the optimal statistic. We apply the criteria to two-sided problems; combining independent tests; testing that the mean of a spherical normal distribution is 0, and extensions to other spherically symmetric and exponential distributions; Bartlett's problem of testing the equality of several normal variances; and testing for one outlier in a normal linear model. For the most part, the optimal statistic is quite easy to use. Often, but not always, it is the likelihood ratio statistic.