Iterative aggregation of crowdsourced tasks within the belief function theory

Abstract

With the growing of crowdsourcing services, gathering training data for supervised machine learning has become cheaper and faster than engaging experts. However, the quality of the crowd-generated labels remains an open issue. This is basically due to the wide ranging expertise levels of the participants in the labeling process. In this paper, we present an iterative approach of label aggregation based on the belief function theory that simultanously estimates labels, the reliability of participants and difficulty of each task. Our empirical evaluation demonstrate the efficiency of our method as it gives better quality labels.