Theory and Practice in Nonprobability Surveys

Abstract

Many in the survey research community have expressed concern at the growing popularity of nonprobability surveys. The absence of random selection prompts justified concerns about self-selection producing biased results and means that traditional, design-based estimation is inappropriate. This paper seeks to provide insight into the conditions under which nonprobability surveys can be expected to provide estimates free of selection bias. In fields such as epidemiology and economics that routinely work with observational data, researchers have identified the necessary conditions for unbiased estimation of causal effects when treatments are not assigned randomly. Similar conditions apply to survey estimates when respondents are not randomly selected. Drawing on this body of research, we propose a framework composed of three elements that determine the level of selection bias in survey estimates. In this paper, we first provide a general overview of these components and demonstrate the link between causal inference and survey inference in the probability-based setting. Second, we give simplified examples to demonstrate how each of the components can contribute to bias in survey estimates. Finally, we review current practices in the area of nonprobability data collection and estimation, and specify how these methods relate to the elements identified here.