Vision Based Hand Puppet

Abstract

The aim of this project is to develop a multimodal interface for a digital puppetry application, which is suitable for creative collaboration of multiple performers. This is achieved by manipulating the low- and high level aspects of 3D hierarchical digital models in real–time. In particular, the hands and the face of multiple performers are tracked in order to recognize their gestures and facial expressions, which are then mapped to kinematic parameters of digital puppets. The visualization of the puppet is provided as a feedback for the performer, and as an entertainment medium for the audience. Possible uses of this system include digital theaters, simplified animation tools, remote fullbody interaction and sign-language visualization. The application consists of two separate hand tracking modules aimed at different shape and motion parameters, a facial feature tracker, a hand gesture and facial expression classification module, an XML based lowbandwidth communication module and a visualization module capable of handling inverse kinematics and skeletal animation. The methods employed do not depend on special hardware and do not require high computational power, as each module runs on separate computers.