A gold standards-based crowd label aggregation within the belief function theory

Abstract

Crowdsourcing, in particular microtasking is now a powerful concept used by employers in order to obtain answers on tasks hardly handled by automated computation. These answers are provided by human employees and then combined to get a final answer. Nevertheless, the quality of participants in microtasking platforms is often heterogeneous which makes results imperfect and thus not fully reliable. To tackle this problem, we propose a new approach of label aggregation based on gold standards under the belief function theory. This latter provides several tools able to represent and even combine imperfect information. Experiments conducted on both simulated and real world datasets show that our approach improves results quality even with a high ratio of bad workers.