A novel strategy for active task assignment in crowd labeling

Abstract

Active learning strategies are often used in crowd labeling to improve the task assignment. However, these strategies, which evaluates each possible assignment at first and then greedily selects the optimal one, may require prohibitive computation time but still cannot improve the assignment to the utmost. Thus, we develop a novel strategy by firstly deriving an efficient algorithm for assignment evaluation. Then, to overcome the uncertainty of labels, we modulate the scope of the greedy task assignment with the posterior uncertainty and keep the evaluation being optimistic. The experiments on four popular worker models and four MTurk datasets show that our strategy achieves the best performance and highest computation efficiency.