Learning from crowds and experts

Abstract

Crowdsourcing services are often used to collect a large amount of labeled data for machine learning. Although they provide us an easy way to get labels at very low cost in a short period, they have serious limitations. One of them is the variable quality of the crowd-generated data. There have been many attempts to increase the reliability of crowdgenerated data and the quality of classifiers obtained from such data. However, in these problem settings, relatively few researchers have tried using expert-generated data to achieve further improvements. In this paper, we apply three models that deal with the problem of learning from crowds to this problem: a latent class model, a personal classifier model, and a data-dependent error model. We evaluate these methods against two baseline methods on a real data set to demonstrate the effectiveness of combining crowd-generated data and expert-generated data.