Statistical power in the journal of research in music education (2000-2010): A retrospective power analysis

Abstract

During the last half of the 20th century, researchers became increasingly aware of a tendency in much social science research to lack sufficient statistical power. Low power can cause studies to be unable to reliably detect true effects in a population (Type II error) and to overestimate the practical significance of small effects when they are detected. Despite cautionary writings and the widespread availability of helpful computational tools, systemically underpowered research persists in the majority of contemporary academic journals. While the Journal of Research in Music Education is arguably the profession's most prominent platform for empirical research, the degree to which this issue may be present has not yet been examined. Results found that the research articles published in the journal between 2000-2010 share similarities with other prominent journals in education and educational psychology in that, on average, the statistical power of the designs fell well below commonly accepted thresholds. Survey research in music education gener-ally fared the best, being more adequately powered (.72) than many other academic disciplines, although still below the de facto standard of .80. By contrast, the average power of designs involv-ing the manipulation of experimental conditions was much lower (.55). These figures represent best-case scenarios. Reports often did not contain all relevant information necessary to determine some common hidden drains to power, requiring some concessions to be made during calculation that likely overestimate the final results. Suggestions for maximizing statistical power are offered as well as a discussion of alternatives to classical null hypotheses significance testing.