Pay Rates and Subject Performance in Social Science Experiments Using Crowdsourced Online Samples

Abstract

Mechanical Turk has become an important source of subjects for social science experiments, providing a low-cost alternative to the convenience of using undergraduates while avoiding the expense of drawing fully representative samples. However, we know little about how the rates we pay to Turkers for participating in social science experiments affects their participation. This study examines subject performance using two experiments - a short survey experiment and a longer dynamic process tracing study of political campaigns - that recruited Turkers at different rates of pay. Looking at demographics and using measures of attention, engagement and evaluation of the candidates, we find no effects of pay rates upon subject recruitment or participation. We conclude by discussing implications and ethical standards of pay.