Hotspot components for gesture-based interaction

Abstract

We present a novel camera-based adaptable user interface system that uses hotspot components for 2D gesture-based interaction. A camera points to the desktop and the image captured by the camera appears on the user's screen. A hotspot area is activated when a user's hand passes through the rectangle that defines it. For example, a right_left hotspot activates when the user moves his hand from right to left, entering and exiting the rectangle. Our system is highly flexible because it allows the user to customize the interface as follows: (1) hotspot areas can be created anywhere within the camera-captured image; (2) new commands can be assigned to particular hotspots or composite hotspot sequences (e.g., right_Left for previous webpage; up+right+down for webpage reload); (3) a physical workspace on the desktop can be defined by pointing the camera to any location; (4) different hotspot layouts for different applications can be created and saved. The system works with an inexpensive webcam in real time and uses machine learning to automatically detect skin areas for robust gesture recognition. © IFIP International Federation for Information Processing 2005.