SmartCrowd: A Workflow Framework for Complex Crowdsourcing Tasks

Abstract

Over the past decade, a number of frameworks have been introduced to support different crowdsourcing tasks. However, complex creative tasks have remained out of reach for workflow modeling. Unlike typical tasks, creative tasks are often interdependent, requiring human cognitive ability and team collaboration. The crowd workers are required not only to perform typical tasks, but also to participate in the analysis and manipulation of complex tasks, hence the number and execution order of tasks are unknown until runtime. Thus, it is difficult to model this kind of complex tasks by using existing workflow approaches. Therefore, we propose a workflow modeling approach based on state machine to design crowdsourcing model that can be translated into SCXML code and executed by an open source engine. This approach and engine are embodied in SmartCrowd. Through two evaluations, we found that SmartCrowd can provide support for complex crowdsourcing tasks, especially on creative tasks. Moreover, we introduce a set of basic design patterns, and by employing them to compose complex patterns, our framework can support more crowdsourcing research.