A Tale of Two Sample Sources: Do Results from Online Panel Data and Conventional Data Converge?

Abstract

Samples drawn from commercial online panel data (OPD) are becoming more prevalent in applied psychology research, but they remain controversial due to concerns with data quality. In order to examine the validity of OPD, we conduct meta-analyses of online panel samples and compare internal reliability estimates for scales and effect size estimates for IV–DV relations commonly found in the field with those based on conventionally sourced data. Results based on 90 independent samples and 32,121 participants show OPD has similar psychometric properties and produces criterion validities that generally fall within the credibility intervals of existing meta-analytic results from conventionally sourced data. We suggest that, with appropriate caution, OPD are suitable for many exploratory research questions in the field of applied psychology.