A valid and reliable measure of nothing: disentangling the ‘‘Gavagai effect’’ in survey data

Abstract

Background. In three recent studies, Maul demonstrated that sets of nonsense items can acquire excellent psychometric properties. Our aim was to find out why responses to nonsense items acquire a well-defined structure and high internal consistency. Method. We designed two studies. In the first study, 610 participants responded to eight items where the central term (intelligence) was replaced by the term ‘‘gavagai’’. In the second study, 548 participants responded to seven items whose content was totally invented. We asked the participants if they gave any meaning to ‘‘gavagai’’, and conducted analyses aimed at uncovering the most suitable structure for modeling responses to meaningless items. Results. In the first study, 81.3% of the sample gave ‘‘gavagai’’ meaning, while 18.7% showed they had given it no interpretation. The factorial structures of the two groups were very different from each other. In the second study, the factorial model fitted almost perfectly. However, further analysis revealed that the structure of the data was not continuous but categorical with three unordered classes very similar to midpoint, disacquiescent, and random response styles. Discussion. Apparently good psychometric properties on meaningless scales may be due to (a) respondents actually giving an interpretation to the item and responding according to that interpretation, or (b) a false positive because the statistical fit of the factorial model is not sensitive to cases where the actual structure of the data does not come from a common factor. In conclusion, the problem is not in factor analysis, but in the ability of the researcher to elaborate substantive hypotheses about the structure of the data, to employ analytical procedures congruent with those hypotheses, and to understand that a good fit in factor analysis does not have a univocal interpretation and is not sufficient evidence of either validity nor good psychometric properties.