Odd Leaf Out
Improving visual recognition with games

Derek L. Hansen*, David W. Jacobs^, Darcy Lewis*, Arijit Biswas^, Jennifer Preece*, Dana Rotman*, Eric Stevens^
*College of Information Studies (iSchool) and Human-Computer Interaction Lab, ^Department of Computer Science
University of Maryland, College Park, Maryland, USA


Abstract—A growing number of projects are solving complex
computational and scientific tasks by soliciting human feedback
through games. Many games with a purpose focus on generating
textual tags for images. In contrast, we introduce a new game,
Odd Leaf Out, which provides players with an enjoyable and
educational game that serves the purpose of identifying
misclassification errors in a large database of labeled leaf images.
The game uses a novel mechanism to solicit useful information
from players’ incorrect answers. A study of 165 players showed
that game data can be used to identify mislabeled leaves much
more quickly than would have been possible using a computer
vision algorithm alone. Domain novices and experts were equally
good at identifying mislabeled images, although domain experts
enjoyed the game more. We discuss the successes and challenges
of this new game, which can be applied to other domains with
labeled image datasets.
Keywords-games with a purpose, computer vision, error
detection, leaf identification
I. INTRODUCTION
A growing number of scientific projects use images that are
created and curated through crowdsourcing. Flickr users submit
images of rare species to the Encyclopedia of Life (EOL);
space enthusiasts classify Hubble images of galaxies at Galaxy
Zoo; and citizen scientists use mobile apps to submit species’
photos to online conservation projects such as Project Noah
and iNaturalist. Having volunteer enthusiasts collect and
classify images helps to tap into enormous reserves of potential
human power [1]. It also inevitably introduces classification
errors into the underlying datasets, a problem evident in even
expertly curated image datasets. Catching a handful of
misclassified images in a large corpus of data can be a tedious,
time-intensive, and costly process. Automated image-
classification algorithms can help catch some of the most
egregious errors, but fail to capture more subtle errors that
human expertise can uncover. Unfortunately, image
classification by humans introduces its own set of problems. In
this paper we propose a novel game, Odd Leaf Out, which
melds human expertise with computer vision algorithms to help
identify misclassified images.
Odd Leaf Out was inspired by other “games with a
purpose” (GWAP) [2] that make laborious tasks enjoyable by
recasting them as games. In this case, the laborious task is
identifying misclassified images. We have constructed an
initial dataset of leaf images tagged with their associated plant
species, and intend to create a much larger leaf dataset using
experts and citizen scientists. The dataset will be used by
Leafsnap, an open-access and mobile image leaf recognition
system used by lay-people for species discovery. Assuring the
accuracy of the dataset is essential to the project’s success.
While the Odd Leaf Out game is specifically focused on
identifying leaf image classification errors, its novel game
design can be used to find errors in other image datasets used in
various scientific endeavors. For example, it could help find
errors in image corpora of other biological species (butterflies,
bacteria), astronomical phenomena (galaxies, stars), human
faces and emotions, and even abstract visual representation of
other scientific phenomena (protein structures).
The major contributions of this paper include the novel
game mechanics of Odd Leaf Out, an evaluation of its
enjoyability, and an assessment of its effectiveness for
identifying errors in a dataset. We begin by reviewing the
literature on games with a purpose and visual recognition
algorithms. Next, we explain the Odd Leaf Out game
mechanics and goals, followed by a description of our
evaluation methods and results. Finally, we discuss the
implications and future possibilities that our work inspires.
II. PREVIOUS WORK
A. Computer Vision
The availability of Internet resources has fueled the rapid
accumulation of large datasets of labeled images for use in
computer vision. For example, the LabelMe [3] project
provides web-based tools for labeling objects and regions in
images, along with over 10,000 labeled images. Researchers
have used Amazon’s Mechanical Turk (MTurk) to acquire
over 10,000 labels that describe attributes of face images, such
as gender, ethnicity, or facial expression [4]. These are used to
build attribute detectors for a face search engine. As [5] note,
annotation errors are inevitable in large-scale annotation
projects. They suggest some strategies for dealing with these
problems, such as having MTurk workers annotate some
images with known, ground truth annotations, to assess the
accuracy of individual workers. However this does not
completely remove mislabeling due to random mistakes
(clicking on the wrong button by accident), lapses in
judgment, or uncertainty in difficult cases. They also suggest
obtaining redundant annotations, for example, producing
annotations based on the two out of three annotators that are
most consistent, as a potential solution. However, this can
triple the effort required for annotation. Games that can
Funding provided by NSF SoCS Grant #SES-0968546
2011 IEEE International Conference on Privacy, Security, Risk, and Trust, and IEEE International Conference on Social Computing
978-0-7695-4578-3/11 $26.00 © 2011 IEEE
DOI
87