Low-Cost Multi-Touch Sensing through Frustrated Total Internal Refl ection Jefferson Y. Han Media Research Laboratory New York University 719 Broadway. New York, NY 10003 E-mail: jhan@mrl.nyu.edu ABSTRACT This paper describes a simple, inexpensive, and scalable technique for enabling high-resolution multi-touch sensing on rear-projected interactive surfaces based on frustrated total internal refl ection. We review previous applications of this phenomenon to sensing, provide implementation details, discuss results from our initial prototype, and outline future directions. ACM Classifi cation: H.5.2 [User Interfaces]: Input Devices and Strategies General Terms: Human Factors Keywords: multi-touch, touch, tactile, frustrated total internal refl ection INTRODUCTION While touch sensing is commonplace for single points of contact, it is still diffi cult and/or expensive to construct a touch sensor that can register multiple simultaneous points of contact. Multi-touch sensing enables a user to interact with a system with more than one fi nger at a time, as in chording and bi-manual operations. Such sensing devices are inherently also able to accommodate multiple users simultaneously, which is especially useful for larger shared-display systems such as interactive walls and tabletops. Initial investigations, though sparse due to the prohibitive availability of these devices, nonetheless reveal exciting potential for novel interaction techniques [1][2][11][12][19][23][26][27]. We present a simple technique for robust multi-touch sensing at a minimum of engineering effort and expense. It is based on frustrated total internal refl ection (FTIR), a phenomenon familiar to both the biometric and robot sensing communities. It acquires true touch image information at high spatial and temporal resolutions, is scalable to large installations, and is well suited for use with rear-projection. It is not the aim of this paper to explore the multi-touch interaction techniques that this system enables, but rather to make the technology readily available to those who wish to do so. RELATED WORK A straightforward approach to multi-touch sensing is to simply utilize a plurality of discrete sensors, making an individual connection to each sensor as in the Tactex MTC Express [20]. They can also be arranged in a matrix confi guration with some active element (e.g. diode, transistor) at each node, as in the device featured in Lee et al.���s seminal work [11], and also in Westerman and Elias���s commercial FingerWorks iGesturePad [3][22]. Through careful driving techniques, it is possible to gather multi-touch information from a purely passive matrix of force-sensitive-resistors (FSRs) as developed by Hillis [6], or capacitive electrodes, such as in [18] and the recent SmartSkin [19], and thus achieve a great reduction in complexity. However, these devices still require very many connections, which keeps their resolution limited in practice (under 100��100). Furthermore, these systems are visually opaque, forcing systems to resort to top-projection for integration with a graphic display. Alternatively, video cameras present a very convenient way to acquire high-resolution datasets at rapid rates, and thus have naturally been explored for touch sensing. Recent approaches include estimating depth from intensity as in HoloWall [15], estimating depth from stereo as in TouchLight [26] and the Visual Touchpad [12], and tracking markers embedded within a deformable substrate as in GelForce [8]. Figure 1: Simple examples of multi-touch interaction using our FTIR technique Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profi t or commercial advantage and that copies bear this notice and the full citation on the fi rst page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specifi c permission and/or a fee. UIST'05, October 23���27, 2005, Seattle, Washington, USA. Copyright 2005 ACM 1-59593-023-X/05/0010...$5.00. 115

FTIR SENSING TECHNIQUES An interesting group of techniques are those that make use of frustrated total internal refl ection (FTIR). When light encounters an interface to a medium with a lower index of refraction (e.g. glass to air), the light becomes refracted to an extent which depends on its angle of incidence, and beyond a certain critical angle, it undergoes total internal refl ection (TIR). Fiber optics, light pipes, and other optical waveguides rely on this phenomenon to transport light effi ciently with very little loss. However, another material at the interface can frustrate this total internal refl ection, causing light to escape the waveguide there instead. This phenomenon is well known and has been used in the biometrics community to image fi ngerprint ridges since at least the 1960s [25]. The fi rst application to touch input appears to have been disclosed in 1970 in a binary device that detects the attenuation of light through a platen waveguide caused by a fi nger in contact [7]. Mueller exploited the phenomenon in 1973 for an imaging touch sensor that allowed users to ���paint��� onto a display using free-form objects, such as brushes, styli and fi ngers [17]. In that device, light from the fl ying spot of a CRT is totally internally refl ected off the face of a large prism and focused onto a single photodetector, thereby generating an updating bitmap of areas that are being contacted. Greene rediscovered this method in 1985 in his Drawing Prism [5], but updated in optically inverted confi guration, with a video camera and a broad light source replacing the CRT and photodetector. Mallos disclosed a CRT-based touch sensor in 1981 which replaces the bulky prism with a thin platen waveguide [13], and operates by detecting the light scattered away by an object in optical contact. Some more recent fi ngerprint sensors take this approach as well [4]. The robotics community has also used this approach since 1984 in the construction of tactile sensors for robot grippers, but with a compliant surface overlay [27][16][23]. This is a structured fl exible membrane which is normally kept separate by an air-gap, but when depressed, makes optical contact with the waveguide. This effectively makes the sensor responsive to force rather than contact. Kasday proposes a similar modifi cation [9] to the Mallos sensor. IMPLEMENTATION Though these FTIR techniques have fallen out of usage, modern-day accessibility to machine vision hardware and processing makes a compelling case to revisit them. For multi-touch sensing, we adapt the Mallos/Kasday design, but in its dual confi guration, with the optical paths reversed. Alternatively, it can be thought of as a FTIR fi ngerprint sensor, or a FTIR robot tactile sensor, only greatly scaled up. In our prototype, we use a 16���x12��� (406mm x 305mm), ����� (6.4mm) thick sheet of acrylic, whose edges have been polished clear, as an optical waveguide. Common glass is unsuitable here due to its poor optical transmittance however we have also used clearer glass formulations (e.g. ���water white���) successfully. Though more expensive, such glass is structurally stiffer, and is far less easily scratched than acrylic. This sheet is edge-lit by high-power infrared LEDs, which are placed directly against the polished edges so as maximize coupling into total internal refl ection (total optical power: 460mW @ 880nm), while a digital video camera equipped with a matching band-pass fi lter is mounted orthogonally. TIR keeps the light trapped within the sheet, except at points where it is frustrated by some object (e.g. fi nger) in optical contact, causing light to scatter out through the sheet towards the camera (see Figure 3). Only simple image processing operations (rectifi cation, background subtraction, noise removal, and connected components analysis) are required for each frame, while routine machine vision tracking techniques are used to interpret the sequences into discrete touch events and strokes. Video is captured at 8-bit monochrome at 30fps at a resolution of 640x480 (corresponding to 1mm2 precision on the surface) all processing is easily performed in real-time by a modest 2GHz Pentium 4 workstation. Our technique provides full imaging touch information without occlusion or ambiguity issues. The touch sense is zero-force and true: it accurately discriminates touch from a very slight hover. It samples at both high temporal and spatial resolutions. Pressure is not sensed, though this is largely mitigated by the precision with which it can determine the contact area of a depressed fi nger. It is inexpensive to construct, and trivially scalable to much larger surfaces- the Figure 2: Some previous applications of FTIR to sensing: disclosures by (a) White (b) Johnson and Fryberger (c) Mueller (d) Mallos (e) Kasday 116