Aggregating Human-Expert Opinions for Multi-Label Classification

Abstract

This paper introduces a multi-label classification problem to the field of human computation. The problem involves training data such that each instance belongs to a set of classes. The true class sets of all the instances are provided together with their estimations presented by m human experts. Given the training data and the class-set estimates of the m experts for a new instance, the multi-label classification problem is to estimate the true class set of that instance. To solve the problem we propose an ensemble approach. Experiments show that the approach can outperform the best expert and the majority vote of the experts.