On the Statistical Analysis of Experiments With Manipulation Checks

Abstract

Psychologists are often interested in the effect of an internal state, such as ego depletion, that cannot be directly assigned in an experiment. Instead, they assign participants to a manipulation intended to produce this state and use manipulation checks to assess the manipulation’s effectiveness. In this article, I discuss statistical analyses for experiments in which researchers are primarily interested in the average treatment effect (ATE) of the target internal state rather than that of the manipulation. Often, researchers estimate the association of the manipulation itself with the dependent variable, but this intention-to-treat (ITT) estimator is typically biased for the ATE of the target state, and the bias could be either toward the null (conservative) or away from the null. I discuss the fairly stringent assumptions under which this estimator is conservative. Given this, I argue against the status-quo practice of interpreting the ITT estimate as the effect of the target state without any explicit discussion of whether these assumptions hold. Under a somewhat weaker version of the same assumptions, one can alternatively use instrumental-variables (IVs) analysis to directly estimate the effect of the target state. IVs analysis complements ITT analysis by directly addressing the central question of interest. As a running example, I consider a multisite replication study on the ego-depletion effect, in which the manipulation’s partial effectiveness led to criticism and several reanalyses that arrived at varying conclusions. I use IVs analysis to directly account for the manipulation’s partial effectiveness; this corroborated the replication authors’ reported null results.