Using randomization tests to assess treatment effects in multiple-group interrupted time series analysis

Abstract

Rationale, aims, and objectives: Interrupted time series analysis (ITSA) is a popular evaluation methodology in which a single treatment unit's outcome is studied over time and the intervention is expected to “interrupt” the level and/or trend of the outcome, subsequent to its introduction. The internal validity of this analysis is strengthened considerably if the treated unit is contrasted with a comparable control group. However, multiple-group ITSA typically has small sample sizes, and parametric methods for multiple-group ITSA require strong assumptions that are unlikely to be met, possibly resulting in misleading P values. In this paper, randomization tests are introduced as a non-parametric, distribution-free option for computing exact P values. Method: The effect of California's Proposition 99 (passed in 1988) for reducing cigarette sales is evaluated by comparing California (CA) to Montana (MT) and Idaho (ID)—the two best matched control states not exposed to any smoking reduction initiatives. Results from randomization tests are contrasted to those of interrupted time series analysis regression (ITSAREG)—a commonly used parametric approach for evaluating treatment effects in ITSA studies. Results: Both approaches found ID and MT to be comparable to CA on their preintervention time series, and both approaches equally found CA to have statistically lower cigarette sales in the postintervention period (P < 0.01). Conclusions: In these data, randomization tests computed P values comparable with ITSAREG, bolstering confidence in the intervention effect. Routinely including randomization tests as a complement, or alternative, to parametric methods is therefore beneficial because randomization tests are free of assumptions regarding sample size and distribution and are extremely flexible in the choice of test statistic.