Collecting Online Survey Data: A Comparison of Data Quality among a Commercial Panel & MTurk

Abstract

Scholars seeking online data collection have sought the assistance of professional panels, which are commercial entities that recruit prospective research participants. Although not intended for recruiting participants, Amazon’s Mechanical Turk (MTurk) service is a crowdsourcing platform that is being utilized for a lower cost. This study compares data collected from MTurk (N = 760) and a professional panel (N = 1,110) to assess aspects of data quality and respondent features. Results show that MTurk may produce better data quality based on completion rate and success in passing manipulation checks. Participants also differ in terms of educational attainment, age, and preferences of certain devices used for participation. Overall, results provide insight that researchers collecting online data should take into consideration before choosing a data collection platform.