Detecting Fraud in Online Surveys by Tracing, Scoring, and Visualizing IP Addresses

Abstract

Amazon’s Mechanical Turk (MTurk) and other online convenience samples are used in thousands of published social science studies every year. One survey estimated that, in 2015 alone, over 1,200 published studies used MTurk (Bohannon, 2016). Another found that over 40% of studies in two top psychology journals in 2015 included at least one MTurk experiment (Zhou & Fishbach, 2016). Many recent studies have validated the use of MTurk to address substantive questions of interest in the social sciences, e.g., (Clifford, Jewell, & Waggoner, 2015), (Huff & Tingley, 2015), (Casler, Bickel, & Hackett, 2013), (Buhrmester, Kwang, & Gosling, 2011). Because of this, recent reports of widespread fraudulent responses on MTurk, up to 25% of respondents in some studies, set off a panic in academia (Dreyfuss, 2018), (Ahler, Roush, & Sood, 2018). The problem has been traced to the use of Virtual Private Servers (VPS) to answer U.S. surveys from abroad (Dennis, Goodson, & Pearson, 2018), (TurkPrime, 2018), and may have affected studies as far back as 2015 (Kennedy, Clifford, Burleigh, Jewell, & Waggoner, 2018). Yet, the tools available to social scientists to check their surveys for VPS use and non-U.S. respndents are not easily usable for most researchers; some are outdated and involve Python programming (Ahler et al., 2018), while other require researchers to paste IP Addresses in one at a time (Dennis et al., 2018). As more research moves online using services like MTurk, CrowdFlower, and Luc.id, there is a need for tools to check IP addresses that fit into standard social science research flow.