Giving less power to statistical power

Abstract

Researchers often need to justify their choice of sample size, particularly in fields such as animal and clinical research, where there are obvious ethical concerns about relying on too many or too few study subjects. The common approach is still to depend on statistical power calculations, typically carried out using simple formulas and default values. Over-reliance on power, however, not only carries the baggage of statistical hypothesis tests that have been criticized for decades, but also blocks an opportunity to strengthen the research in the design phase by learning about challenges in interpretation before the study is carried out. We recommend constructing a ‘quantitative backdrop’ in the planning stage of a study, which means explicitly connecting ranges of possible research outcomes to their expected real-life implications. Such a backdrop can facilitate a priori considerations of how potential results, for example represented by intervals, will ultimately be interpreted. It can also serve, in principle, to help select single values of interest for use in traditional power analyses, or, better, inform sample size investigations based on the goal of achieving an interval width narrow enough to distinguish values deemed practically or clinically important from those not representing practically meaningful effects. The latter bases calculations on a desired precision, rather than desired power. Sample size justification should not be seen as an automatic math exercise with a right answer, but as a nuanced a priori investigation of measurement, design, analysis and interpretation challenges. Construction of the quantitative backdrop provides a tangible starting place for such an investigative process.