Hand Distinction for Multi-Touch Tabletop Interaction Chi Tai Dang, Martin Straub, Elisabeth Andr�� e University of Augsburg Institute of Computer Science Multimedia Concepts and their Applications Universitaetsstr. 6a, 86159 Augsburg, Germany {dang, andre}@informatik.uni-augsburg.de, martin.k.straub@googlemail.com ABSTRACT Recent multi-touch multi-user tabletop systems offer rich touch contact properties to applications. Not only they pro- vide touch positions, but also finger orientations. Applica- tions can use these properties separated for each finger or derive information by combining the given touch contact data. In this paper, we present an approach to map fingers to their associated joined hand contributing to potential en- hancements for gesture recognition and user interaction. For instance, a gesture can be composed of multiple fingers of one hand or different hands. Therefore, we present a sim- ple heuristic for mapping fingers to hands that makes use of constraints applied to the touch position combined with the finger orientation. We tested our approach with collected di- verse touch contact data and analyze the results. Author Keywords Input/Interaction, Multi-Touch, Tracking, Touch Properties, Tabletop Hardware. ACM Classification Keywords H.5.2 [Information Interfaces and Presentation]: User Inter- faces���Input devices and strategies I.4.9 [Image Processing and Computer Vision]: Applications INTRODUCTION Multi-touch is an increasingly emerging user interaction tech- nology found in small display devices, such as the Apple iPhone, recent multi-touch notebooks as well as in larger form factors, such as the Microsoft Surface1. While the sen- sor capabilities of small display devices limit the kind of de- tectable touch properties, camera based systems as used for most interactive tabletop surfaces are more abundant and fa- cilitate the detection of rich touch contact properties. These interactive tabletop surfaces have become widespread over the last years due to a cost-effective and reliable construction 1http://www.surface.com Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. ITS ���09, November 23-25 2009, Banff, Alberta, Canada Copyright 2009 978-1-60558-733-2/09/11... $10.00 using Han���s FTIR-approach [12] or another popular technol- ogy called diffused illumination as utilized in [19]. Along with the tabletop devices, developers create applications that make use of touch interaction. Examples include dual finger selection techniques to allow for greater precision at point- ing tasks [4, 23, 3, 13], techniques to emulate graphical com- mands, such as deleting or moving groups of objects by us- ing the complete hand [10, 22], techniques for fluid inter- action, such as integrated control of rotation and translation [14], or techniques involving several fingers to emulate the functionality of computer mice [20, 17]. The easiest tech- nique to recognize user interaction is to consider touch con- tacts without respect to their joined hand. However, on table- top devices, user input can originate from different hands of one or several users and from multiple fingers of their hands. Applications that are able to determine whether user input is provided by one or several hands may exploit the potential of a more natural and richer repertoire of input gestures. Often, a repertoire of gestures can only be extended at the expense of lower robustness, since new gestures distinguish from ex- isting ones only by subtle variations. The inclusion of two- handed gestures would, however, not result into a significant loss of accuracy because information on handedness may be employed as a highly discriminative feature to classify input. In applications that emulate the functions of mouse buttons, techniques to distinguish one-handed from two-handed in- put could, for example, ensure that mouse actions are only triggered if touch points come from one hand and not from different hands that accidentally form a similar constellation. Moscovich et. al. [18] present a study, which demonstrates that one-handed input and two-handed input is not interre- placeable, since handedness has an impact on the ease and accuracy with which gestures may be conducted. In par- ticular, they showed that one-handed input is suitable for moving, stretching or rotating an object while two hands are more decent for tasks wherein separate control of points, such as selecting a region, is required. Their study advocates the use of both unimanual and bimanual gestures depending on the accuracy with which certain tasks can be performed. Here, distinguishing hands helps adapt the application be- havior to offer possibilities to ease precise tasks in case of two hands, for example by changing the resolution or speed of object movements. Even though many research projects exploit the potential of a 101

richer repertoire of input gestures resulting from two-handed input, current work usually does not consider cases where a collection of touch points should be interpreted differently depending on whether they originate from the finger of one hand or the fingers of several hands. For example, Benko and colleagues [4] present a dual finger stretch technique where one finger is used to select an area of the interface and a sec- ond one is used to scale the area. However, when evaluating their gestures, they start from the assumption that all ges- tures are executed with two fingers of two different hands belonging to a particular person. Such a gesture could also be performed by two fingers of the same hand as depicted to the left in figure 1. As a consequence, it might be difficult to Figure 1. Gestures with one hand (left) and two hands (right) creating the same touch positions and tracks. distinguish a dual finger stretch gesture performed by fingers of the same hand from a dual hand duplicate gesture where a copy of a visual object is produced by pulling the object apart with the fingers of two separate hands. Another exam- ple pertains to the application behavior for multiple objects when two objects are moved together. A one-handed gesture could result in merging those objects whereas a two-handed gesture would overlay one object with the other for a precise comparison task. Peltonen and colleagues [19] report on problems that arise when multiple users interact in parallel on large surfaces and unintentionally break territorial boundaries. Techniques that distinguish between one-handed and multi-handed input may help resolve such conflicts. Consider, for example, the case where a photo is accidentally zoomed because two users are trying to move it toward themselves as depicted in figure 2. If a system is, however, able to recognize that two users Figure 2. Two different users executing a pulling away gesture. are manipulating one and the same object in parallel, but with different intentions, it might respond to the users��� be- havior in a more appropriate manner. For example, it might lock the object until the users have agreed upon what to do with the object or create a duplicate of the object. Terrenghi and colleagues [21] conducted a study to investi- gate whether people manipulate physical objects on a table- top differently than digital objects. They found that users tended to use just one hand when interacting with digital ob- jects. Distinguishing hands and offering interaction based on handedness might encourage users to use both hands. For example, participants were requested to spatially structure objects. In the physical tasks, users could move multiple pieces altogether to form a structure, but it was hard to im- itate such an action in the digital tasks. However, a hand distinction could enable an adaption of the application be- havior, so as to select all underlying and overlying objects for movement if both hands are touching an object, whereas one hand selects only the directly touched object. In this paper, we contribute to potential enhancements of gesture recognition by explicitly distinguishing one-handed from two-handed gestures. Tabletop tracking components such as used in the Microsoft Surface provide a lot of touch properties, but almost no hand related information to appli- cations. Therefore, we present an algorithm that maps touch contacts onto hands with a high level of precision and dis- cuss observations from an empirical evaluation. RELATED WORK Various attempts have been made to distinguish one-handed touches from two-handed touches. The DiamondTouch [5] table makes use of modulated electric fields. When a user touches the surface of the table, the contact areas are capac- itively coupled through the user to a receiver corresponding to that user. In this way, the DiamondTouch table is able to determine for each touch point to which user it belongs. However, DiamondTouch table is not able to distinguish be- tween one- and two-handed gestures from one and the same user. Echtler and colleagues [8] present an approach for FTIR- tables that features the possibility to map finger touches to the hands of a single or several users. It is based on a top- mounted infrared light source that let the hands and arms throw shadows onto the surface. However, they haven���t ex- plored their setup to explicitly distinguish hands yet. An- other approach was employed by Dohse et al. [7] who en- hanced the interaction on an FTIR-table with a camera placed above the interactive surface. This technology allowed them to track hands by means of computer vision techniques, thus assign a specific touch point to a user or his hand. Malik and colleagues [15] and Do-Lenh et al. [6] present various one-handed and two-handed input techniques for a visual touch pad and an augmented tabletop. Both works distinguish single gestures based on positional information for the fingertips. Furthermore Malik [15] consider finger orientations. In their case, there is no need to disambiguate gestures based on handedness because the chosen gestures 102 The ACM International Conference on Interactive Tabletops and Surfaces 2009