Using Quotas Constructed from Auxiliary Data to Guide the Recruitment of Sites for Randomized Controlled Trials

Abstract

Sampling methods such as Stratified Random Sampling can be used to select representative samples of schools for Randomized Controlled Trials (RCTs) of educational interventions. However, these methods may still yield external validity bias when participation by schools is voluntary and participation decisions are associated with unobserved variables. This paper describes how to establish quotas using auxiliary data to avoid including “too many” schools of a particular type. Simulations were conducted to determine whether adding quotas based on a single auxiliary variable reduces the external validity bias. We found that quotas reduced the external validity bias in the auxiliary variable—without increasing the bias in other variables—at some cost by increasing the number of schools that were recruited to reach to study’s target sample size. The findings suggest that in education RCTs, quotas can help to reduce external validity bias when schools are not required to participate.