(C) ACM 2011. This is the author’s version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version will be published in Conference Proceedings of IHM’11, the 23th ACM International Conference of the Association Francophone d'Interaction Homme-Machine, ACM New York, NY, USA, (Nice, October 2011).

Using the User s Point of View for Interaction on Mobile Devices Jérémie Francone, Laurence Nigay Université Joseph Fourier Grenoble 1 Laboratoire d Informatique de Grenoble LIG UMR 5217, Grenoble, F-38041, France {jeremie.francone, laurence.nigay}@imag.fr

RESUME Nous présentons une technique d’interaction pour dispositifs mobiles (smartphone et tablette) basée sur le suivi du visage de l’utilisateur. Cette technique définit de nouvelles possibilités pour l’interaction en entrée et en sortie sur dispositifs mobiles. En sortie, le suivi de la tête peut permettre de contrôler le point de vue sur une scène 3D affichée à l’écran (Head-Coupled Perspective, HCP). Cette technique améliore l’interaction en sortie en offrant la perception de la profondeur et en permettant la visualisation d’un espace de travail plus grand (fenêtre virtuelle). En entrée, le suivi des mouvements de la tête définit une nouvelle modalité d’interaction qui ne requiert pas d’autres capteurs que la caméra du téléphone ou de la tablette. Dans cet article, nous explicitons les possibilités interactionnelles offertes par le suivi de la tête de l’utilisateur sur téléphones ou tablettes, particulièrement adapté au caractère mobile des dispositifs visés. Nous focalisons ensuite sur l’interaction en sortie en présentant plusieurs applications du HCP et en décrivant les résultats d’une expérimentation qualitative sur téléphone et tablette. Mots clés Dispositif Mobile, Modalité d’interaction, Interface 3D, Head-Coupled Perspective. ABSTRACT We study interaction modalities for mobile devices (smartphones and tablets) that rely on a camera-based head tracking. This technique defines new possibilities for input and output interaction. For output, by computing the position of the device according to the user’s head, it is for example possible to realistically control the viewpoint on a 3D scene (Head-Coupled Perspective, HCP). This technique improves the output interaction bandwidth by enhancing the depth perception and by allowing the visualization of large workspaces (virtual window). For input, head movement can be used as a means of interacting with a mobile device. Moreover such an input modality does not require any additional sensor except the built-in front-facing camera. In this paper, we classify the interaction possibilities offered by head tracking on smartphones and tablets. We then focus on the output interaction by introducing several applications of HCP on both
smartphones and tablets and by presenting the results of a qualitative user experiment. Categories and Subject Descriptors H.5.2 [Information Interfaces and Presentation] User Interfaces – Interaction styles. I.3.6 [Computer Graphics] Methodology and Techniques – Interaction techniques. General Terms Design, Human Factors. Keywords Mobile Device, Interaction Modality, 3D Interface, Head-Coupled Perspective. 1. INTRODUCTION Mobile devices such as smartphones and tablets are multifunctional. They enable us to run complex applications including video editors, web browsers and 3D games. As a consequence, more and more commands and data have to be displayed and managed, nearly reaching the processing capacity of a laptop, while the interaction capacity (input/output modalities) is limited due to the small form factor and the various contexts of use (e.g., walking in the street being encumbered by bags). Indeed, the particularity of these devices, that they have to be mobile, implies a limited screen size, no mouse and a lack of physical buttons. Moreover, as the user is potentially in a mobile situation, s/he is not likely to use both hands [18] and can devote only a limited attention to interacting with the application. But as opposed to a laptop, being mobile means that the user can move and the device can be moved, which should be exploited for interaction. Current mobile devices have built-in multi-axis accelerometers, gyroscopes and sometimes a digital compass. These motion sensors are often used for sensing the screen orientation, controlling a character in a game or orienting a map. Since the seminal work of Rekimoto [25], there have been many systems that are based on motion sensors for enriching the interaction capacity of mobile devices. Amongst them, one relevant way to take advantage of the mobility is to use spatially aware displays [13] [14], which transform a position-tracked device into a window on a larger virtual workspace. By moving the device around, the user can access different parts of the workspace. In this paper, we study the use of the spatial relation between the head and the device. In this context, the front-facing camera of the device is used to localize the face with respect to the device. By using vision-based algorithms, we have implemented an efficient
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. IHM’11, October 24-27, 2011, Sophia Antipolis, France Copyright © 2011 ACM 978-1-4503-0822-9/11/10 …$10.00.