The Impact of Insufficient Effort Responding Detection Methods on Substantive Responses: Results from an Experiment Testing Parameter Invariance

Abstract

Researchers using survey methods can choose among a variety of methods aimed at detecting insufficient effort responding among the participants in their studies. Some of these methods entail modifying the survey questionnaire by adding bogus items, instructed-response items, or instructed manipulation checks. While these methods have been found effective in detecting insufficient effort responding, it remains unclear whether their presence in a questionnaire can affect responses to items of substantive research interest. We conducted an experiment investigating this potential impact in a sample (N = 1,092) of working adults. Adopting an invariance testing approach, we assessed whether employing bogus items, instructed-response items, or instructed manipulation checks, with or without warning study participants, respectively, would lead to non-invariant estimates of parameters for substantive variables. Results suggest that, while most parameter estimates were invariant to the use of insufficient effort responding detection methods, the reliability of measurements may be adversely impacted, posing a threat to construct validity and statistical conclusion validity. However, reliability might be maintained when participants are warned at the beginning of the questionnaire. Results also suggest that bogus items may have some advantages over other methods in terms of parameter invariance. We discuss the implications of our findings.