Multimodal interaction for real and virtual environments

Abstract

Multimodal interfaces can leverage the information from multiple modalities to provide robust and error-free interaction. Early multimodal interfaces demonstrate the feasibility of building such systems but focused on specific applications. The challenge in building adaptive systems is lack of techniques for input data fusion. In this direction, we have developed a multimodal head and eye gaze interface and evaluated it in two scenarios. In aviation scenario, our interface has reduced the task time and perceived cognitive load significantly from the existing interface. We have also studied the effect of various output conditions on user's performance in a Virtual Reality (VR) task. Further, we are making our proposed interface to include additional modalities and building novel haptic and multimodal output systems for VR.