A Dynamic Selection Approach for Quality Control Mechanisms in Crowdsourcing

Abstract

Crowdsourcing has emerged as a new model for leveraging human knowledge and intelligence toward accomplishing tasks that are difficult to fulfill effectively with machines alone. However, owing to its open nature, quality control is a big challenge. Current crowdsourcing systems use one or two standard mechanisms for evaluation and quality control of a task, regardless of its type. In this paper, we propose a dynamic approach that exploits task-quality ontology to select the most suitable quality control mechanism (QCM) for a given task based on its type. The proposed approach has been enriched by a reputation engine that collects requesters' feedback on the performance of QCMs. Accordingly, QCMs and tasks were automatically matched using the underlying categorization structure of tasks on one side and the reputation scores of QCMs on the other side. Our experiments establish that our proposed dynamic approach yields better results compared to existing approaches.