Vision-based hand representation and intuitive virtual object manipulation in mixed reality

Abstract

In this paper we introduce a method for representation of the human hand in virtual or mixed environments and interaction with virtual objects. We use a head-mounted display with two cameras installed on top to record the user's hand, which is then cut out and shown in the virtual environment. Extracted features like silhouette and fingertip positions enable direct visual interaction with objects in virtual worlds. Applications of the approach are presented and feature extraction methods, the overlay technique and the hardware- and software setup of the system are described. Finally the benefits and limitations of the approach are discussed. © 2012 by Walter de Gruyter Berlin Boston.