Sing me a song with social significance': The (Mis) use of statistical significance testing in European sociological research

Abstract

In this article, we analyse the ritual of null hypothesis significance testing among the European sociologists. The focus is on the distinction between statistical significance and substantive, sociological significance. We review all articles published in the European Sociological Review between 2000-2004 and 2010-2014 that use regression models (N 356). Our main aim is to determine whether the authors discuss the effect size of their findings and distinguish substantive from statistical significance. We apply a five-item questionnaire to each article and find that about half of the articles erroneously interpret a statistically insignificant coefficient as a zero effect, while only one in three engage in a discussion of the substantive meaning of the effect sizes. Moreover, our findings show a negative trend in the practice of significance testing over the past 15 years. These results are similar to those of the comparable review in the field of economics. We conclude by providing a set of recommendations on how to reduce the misuse of significance testing in sociological research, through an informed benchmarking of the findings.