A Toolkit for Managing Multiple Crowdsourced Top-K Queries

Abstract

Crowdsourced ranking and top-k queries have attracted significant attention recently. Their goal is to combine human cognitive abilities and machine intelligence to rank computer hostile but human friendly items. Many task assignment algorithms and inference approaches have been proposed to publish suitable micro-tasks to the crowd, obtain informative answers, and aggregate the rank from noisy human answers. However, they are all focused on single query processing. To the best of our knowledge, no prior work helps users manage multiple crowdsourced top-k queries. We propose a toolkit, which seamlessly works with most existing inference and task assignment methods, for crowdsourced top-k query management. Our toolkit attempts to optimize human resource allocation and continuously monitors query quality at any stage of the crowdsourcing process. A user can terminate a query early, if the estimated quality already fulfills her requirements. Besides, the toolkit provides user-friendly interfaces for users to initialize queries, monitor execution status, and do more operations by hand.