Limitations of the Sum-and-Alpha Approach to Measurement in Behavioral Research

Abstract

Many behavioral researchers are interested in measuring constructs such as mood, affect, or cognition—all of which cannot be observed directly. Instead, researchers administer items from surveys, tests, or scales to indirectly measure aspects of the construct. Psychometrics is a branch of statistics dedicated to determining whether scores created from item responses are reasonably capturing the intended construct. However, several review papers have shown that behavioral research frequently does not engage with psychometrics and instead creates scores by assigning numerical values to item responses, summing item responses, and reporting reliability of summed scores without any assessment of whether scores are valid. Despite the popularity of this approach, it can limit the informativeness and generalizability of conclusions in behavioral research. The goal of this paper is to raise awareness of these issues with policymakers and other consumers of research and to encourage producers of research to consider readily available, more rigorous approaches.