Phrase detectives: Utilizing collective intelligence for internet-scale language resource creation

Abstract

We are witnessing a paradigm shift in human language technology that may well have an impact on the field comparable to the statistical revolution: acquiring large-scale resources by exploiting collective intelligence. An illustration of this approach is Phrase Detectives, an interactive online game-with-a-purpose for creating anaphorically annotated resources that makes use of a highly distributed population of contributors with different levels of expertise. The paper gives an overview of all aspects of Phrase Detectives, from the design of the game and the methods used, to the results obtained so far. It furthermore summarises the lessons that have been learnt in developing the game to help other researchers assess and implement the approach.