Illusions of significance in a Rugged landscape

Abstract

A great deal of academic research, particularly in the social sciences, makes an implicit assumption that the processes being investigated are largely decomposable, meaning that the effects of individual variables identified in one context are likely to be observed in other contexts as well. Many of the statistical tools employed in analyzing social science data, such as regression and structural equation modeling, implicitly depend on such decomposability. A companion paper (Gill, 2008) proposes that many informing system situations may actually exist on rugged fitness landscapes for which the decomposability assumption is unrealistic. What this paper demonstrates is that applying statistical tests that assume decomposability on rugged landscapes may lead to statistical illusions of significance. These illusions, in turn, may convince the researcher that the decomposability assumption is valid, posing a serious threat to rigor by vastly overstating the statistical significances of observed relationships and by producing spurious significances. These significances, moreover, will not be readily detected by statistical tests that are commonly used to check the validity of the regression assumptions, such as the normality of error terms. This paper also explores the underlying reasons for such illusions and under what circumstances they can be avoided.