The consequences of personality biases in online panels for measuring public opinion

Abstract

Online surveys, particularly those that draw samples from online panels of experienced respondents, now comprise a large segment of the academic and commercial opinion research markets due to their low cost and flexibility. A growing literature examines the implications of online surveys for data quality, most commonly by comparing demographic and political characteristics of different samples. In this paper, we explore the possibility that personality may differentially influence the likelihood of participation in online and face-to-face surveys. We argue that individuals high in extraversion and openness to experience may be underrepresented, and those low in these traits overrepresented, in professionalized online panels given the solitary nature of repeated survey-taking. Since openness to experience in particular is associated with liberal policy positions, differences in this trait may bias estimates of public opinion derived from professionalized online panels. Using data from the 2012 and 2016 dual-mode American National Election Studies, we compare political preferences and personality traits across parallel face-to-face and online samples. Respondents in the online samples were, on average, less open to experience and more politically conservative on a variety of issues compared to their face-to-face counterparts. This was true especially in 2012, when online respondents were drawn from a large panel of experienced respondents. We also find openness to be negatively related to the number of surveys completed by these respondents. These results suggest that reliance on professionalized survey respondents, who comprise the vast majority of online survey samples, can bias estimates of many quantities of interest.