Searching for significance in the scholarship of teaching and learning and finding none: Understanding non-significant results

Abstract

Quantitative results from empirical studies are common in the field of Scholarship of Teaching and Learning (SoTL), but it is important to remain aware of what the results from our studies can, and cannot, tell us. Oftentimes studies conducted to examine teaching and learning are constrained by class size. Small sample sizes negatively influence statistical power and make non-significant results a more likely occurrence. When one finds non-significant results it is important to consider what conclusions can be drawn from the study. This article provides information on null hypothesis significance testing that is relevant to our understanding of non-significant results, and it highlights the importance of recognizing underpowered studies in the teaching and learning literature. Factors that can contribute to non-significant findings in a study are also highlighted. Being aware of these factors, statistical power, and the logic of significance testing will put scholars in a better position to evaluate non-significant results from their own research and that of others.