Opening the Black Box: How to Design Intelligent Decision Support Systems for Crowdsourcing

Abstract

In crowdsourcing, reviewing and evaluating textual data is a latent challenge. While text mining and machine learning represent promising technologies to solve this problem, it is still unclear how information systems based on these technologies (i.e., intelligent decision support systems) should be designed. In this study, we address this gap and develop overarching design requirements , design principles, and design features for intelligent decision support systems in crowdsourcing. The study follows a design science research approach with a cross-industry research consortium comprising 8 organizations. Our results are based on 41 semi-structured interviews, 13 expert workshops with 53 participants, statistical analyses with data from 676 crowdsourcing projects , and 2 field tests. For research, we introduce transparency and control as two additional meta-requirements for intelligent decision support systems and capture seven guiding principles for designing such systems. For practitioners, we describe specific design features that show how to instantiate these principles .