Generalizing from unrepresentative experiments: A stratified propensity score approach

Abstract

The paper addresses means of generalizing from an experiment based on a non-probability sample to a population of interest and to subpopulations of interest, where information is available about relevant covariates in the whole population. Using stratification based on propensity score matching with an external populationwide data set, an estimator of the population average treatment effect is constructed. An example is presented in which the applicability of a major education intervention in a non-probability sample of schools in Texas, USA, is assessed for the state as a whole and for its constituent counties. The implications of the results are discussed for two important situations: how to use this methodology to establish where future experiments should be conducted to improve this generalization and how to construct a priori a strategy for experimentation which will maximize both the initial inferential power and the final inferential basis for a series of experiments. © 2013 Royal Statistical Society.