Guess what? A game for affective annotation of video using crowd sourcing

Abstract

One of the most time consuming and laborious problems facing researchers in Affective Computing is annotation of data, particularly with the recent adoption of multimodal data. Other fields, such as Computer Vision, Language Processing and Information Retrieval have successfully used crowd sourcing (or human computation) games to label their data sets. Inspired by their work, we have developed a Facebook game called Guess What? for labeling multimodal, affective video data. This paper describes the game and an initial evaluation of it for social context labeling. In our experiment, 33 participants used the game to label 154 video/question pairs over the course of a few days, and their overall inter-rater reliability was good (Krippendorff's α = .70). We believe this game will be a useful resource for other researchers and ultimately plan to make Guess What? open source and available to anyone who is interested. © 2011 Springer-Verlag.