When the Human is in the Loop: Cost, Effort and Behavior

Abstract

Data from human-machine interaction can be used to improve the quality of artificial intelligence (AI) systems. When designing a system with humans in the loop, one of the questions to be asked is how much human work is required to create a reliable data collection. Crowdsourcing has become a popular methodology to collect annotations from crowd workers who successfully complete crowdsourcing tasks. Thus, if workers reach varying task completion stages without finally submitting their work, all their effort would be unrewarded and annotations discarded. Task abandonment remains invisible within the platform. On the other hand, paid crowdsourcing dynamics are often influenced by large batches of similar tasks, allowing workers to learn and develop efficient work strategies. To date, however, there is limited research aiming at understanding how human annotators complete tasks over time. Even for complex tasks in lab studies, understanding how behavioral patterns evolve during task completion remains unexplored. The aim of this research is to study how these tasks are completed, to help reduce non-essential data collection costs, and to support workers in efficient task completion. The objective is to reveal what happens with humans in the loop, looking at three related aspects: cost, effort and behavior. In particular, I explore (i) how to make best use of a given budget to conduct data annotation experiments and to collect labelled data; (ii) how the reward received by human workers can be affected by invisible actions in abandoned tasks; and (iii) how they complete tasks to collect the associate reward. I focus on the following research questions (RQs). RQ#1: How can we increase the intrinsic value of tasks? RQ#2: What is the blind effort of workers while completing tasks? RQ#3: What are the patterns displayed by workers as they progress in tasks? Our findings bear implications on building cost-effective human-machine workflows. The proposed methodology has the potential to benefit AI practitioners building human-like products, and to enable domain experts and crowd workers to perform tasks effectively.