Given a large set of data items, we consider the problem of filtering them based on a set of properties that can be verified by humans. This problem is commonplace in crowdsourcing applications, and yet, to our knowledge, no one has considered the formal optimization of this problem. (Typical solutions use heuristics to solve the problem.) We formally state a few different variants of this problem. We develop deterministic and probabilistic algorithms to optimize the expected cost (i.e., number of questions) and expected error. We experimentally show that our algorithms provide definite gains with respect to other strategies. Our algorithms can be applied in a variety of crowdsourcing scenarios and can form an integral part of any query processor that uses human computation. © 2012 ACM.
CITATION STYLE
Parameswaran, A. G., Garcia-Molina, H., Park, H., Polyzotis, N., Ramesh, A., & Widom, J. (2012). CrowdScreen: Algorithms for filtering data with humans. In Proceedings of the ACM SIGMOD International Conference on Management of Data (pp. 361–372). https://doi.org/10.1145/2213836.2213878
Mendeley helps you to discover research relevant for your work.