Magic Music Desk: A multi-modal embodied interactive desk

Abstract

This paper describes a novel multi-modal multi-user audio-visual interface - the Magic Music Desk (MMD) which employs the principles of embodied interaction, and emphasizes social interaction between users. We have developed a novel combination of multiple modalities for the interfaces using speech recognition, hand gesture recognition, sound and visual mixed reality technologies. A new mode of interaction which is called as "What You Say is What You See" (WYSWYS) is demonstrated in our system. This interaction enables all users to visualize each other's spoken words as 3D objects which could be seen by multi-users (this also allows multi-cultural social interaction). © 2003 by Springer Science+Business Media New York.