SignTutor:
An Interactive
System for Sign
Language
Tutoring
Oya Aran, Ismail Ari, Lale Akarun, and Bu¨lent Sankur
Bog˘azic¸i University
Alexandre Benoit and Alice Caplier
GIPSA-Lab
Pavel Campr
University of West Bohemia in Pilsen
Ana Huerta Carrillo
Technical University of Madrid
Franc¸ois-Xavier Fanard
Universite´ Catholique de Louvain
Sign language recognition (SLR) is amultidisciplinary research area involv-ing pattern recognition, computervision, natural language processing,
and linguistics. The concept presents a multifac-
eted problem not only because of the complexity
of the visual analysis of hand gestures, but also
due to the highly multimodal nature of sign
languages. Although sign languages are well-
structured languages with a phonology, mor-
phology, syntax, and grammar, they are different
from spoken languages. The structure of a spo-
ken language makes use of words sequentially,
whereas a sign language makes use of several
body movements in parallel. The linguistic char-
acteristics of sign language are different from
those of spoken languages due to the existence
of several context-affecting components, such
as facial expressions and head movements in
addition to hand movements.1,2
Practice can significantly enhance the learning
of a language when there is validation and feed-
back. This is true for both spoken and sign lan-
guages. For spoken languages, students can
evaluate their own pronunciation and improve
to some extent by listening to themselves. Simi-
larly, sign language teachers suggest that their stu-
dents practice in front of a mirror. An SLR system
lets students practice, validate, and evaluate their
signing. Such systems, including our SignTutor,
can be instrumental in assisting sign language
education, especially for non-native signers.
With our SignTutor system, we aim to teach
the basics of sign language interactively. Sign-
Tutor automatically evaluates the student’s
signing through visual feedback and informa-
tion about the success of the performed sign.
The SignTutor interactive platform enables
users to watch and learn new signs, practice
them, and validate their performance. Sign-
Tutor communicates to students with various
feedback modalities: text message, recorded
video of the user, video of the segmented
hands, or avatar animation. A demonstration
video of SignTutor can be downloaded at
http://www.cmpe.boun.edu.tr/pilab/pilabfiles/
demos/signtutor_demo_DIVX.avi.
Automatic sign language recognition
A brief survey of sign language grammars
illustrates the challenges faced in developing
automated learning and tutoring systems. Sign
language phonology makes use of hand shape,
place of articulation, and movement. The mor-
phology uses directionality, aspect, and numeral
incorporation, while the syntax uses spatial
localization and agreement as well as facial
expressions. The whole message is contained
not only in hand gestures and shapes (manual
signs) but also in facial expressions and head/
shoulder motion (nonmanual signs). As a conse-
quence, the language is intrinsically multimo-
dal. Hence, SLR is a complex task that uses
hand-shape recognition, gesture recognition,
face-and-body-part detection, and facial-
expression recognition as basic building blocks.3
Pioneering research on hand-gesture recogni-
tion and on SLR has mainly used instrumented
gloves, which provide accurate data for hand
position and finger configuration. These systems
require users to wear cumbersome devices on
Feature Article
Language learning
can only advance
with practice and
corrective feedback.
The interactive
system, SignTutor,
evaluates users’
signing and gives
multimodal feedback
to help improve
signing.
1070-986X/09/$25.00 c 2009 IEEE Published by the IEEE Computer Society 81