Iterative reduction worker filtering for crowdsourced label aggregation

Abstract

Quality control has been an important issue in crowdsourcing. In the label collection tasks, for a given question, requesters usually aggregate the redundant answers labeled from multiple workers to obtain the reliable answer. Researchers have proposed various statistical approaches for this crowd label aggregation problem. Intuitively these approaches can generate aggregation results with higher quality if the ability of the set of workers is higher. To select a set of workers who are possible to have the higher ability without additional efforts for the requesters, in contrast to the existing solutions which need to design a proper qualification test or use auxiliary information, we propose an iterative reduction approach for worker filtering by leveraging the similarity of two workers. The worker similarity we select is feasible for the practical cases of incomplete labels. We construct experiments based on both synthetic and real datasets to verify the effectiveness of our approach and discuss the capability of our approach in different cases.