An experimental study of future "natural" multimodal human-computer interaction

Abstract

In order to study users spontaneous formulation of commands in the context of multimodal human-computer interaction (HCI), we conducted a Wizard of Oz experiment on the use of unconstrained speechand 2D-gestures for interacting with standard application software: 8 subjects performedvariousdesignand process control tasks during 3 weekly sessions. Some functionalities of the tnultimodal user interface were simulated by 3 human operators or wizards. First analyses bring out the great diversity of subjects styles and strategies; they also indicate that, in such environments, the addition of spoken natural language to direct manipulation (the manipulation of graphical objects through pointing) improves HCI efficiency and flexibility, whilst command interpretation remains tractable.