Integrating quasi-experimental and inductive designs in evaluation: A case study of the impact of free bus travel on public health

Abstract

Evaluations of ‘natural experiments’ in public policy are typically considered ‘weak’ evidence. Challenges include: making credible claims for causal inference (internal validity); generalizing beyond the case (external validity); and providing useful evidence for decision makers. In public health, where experimental evidence is encouraged by funders and enjoys a degree of rhetorical favour, in theory if not practice, current guidance for evaluating natural experiments focuses largely on methods for strengthening internal validity. Using a case study of the evaluation of free bus travel for young people in London, UK, we demonstrate a pragmatic approach to strengthening both internal and external validity in evaluations through integrating the logic of quasi-experimental methods with inductive qualitative analysis. Combining theoretical and inductive analysis in this way to address questions of policy interest through evaluations of natural experiments may be fruitful, and have methodological advantages over randomized designs.