Collecting survey data with an academic volunteer online panel: representation bias and weighting adjustments

Abstract

Commercial nonprobability online panels are the most common type of panels; however, academics may want to manage their own panels of survey respondents for reasons such as cost, data quality, and methodological transparency. This research addresses the gap in knowledge about non-commercial nonprobability online panels by presenting their recruitment and data collection solutions. The aims of this research are to determine the type and amount of socio-demographic bias in such a panel, and to identify potential solutions to mitigate this bias, including ad-hoc recruitment in each panel wave and using raking as a weighting procedure. The evidence suggests that the most substantial item-level representation bias in such nonprobability samples is for altruistic activities, measured by variables such as voluntary work. This can be explained by the fact that survey participation can be seen as an altruistic activity itself. Also, women, highly educated respondents, and homeowners are notably overrepresented. Importantly, the reviewed solutions proved to be effective in reducing representation bias — including the ad-hoc sample by about 15%, and raking, which included “altruistic” covariates, by up to 28%. However, using raking with multiple covariates came at the expense of an increase in the design effect due to weighting.