Interpretation of gestures and speech: A practical approach to multimodal communication

Abstract

Developing multimodal interfaces is not only a matter of technology. Rather, it implies an adequate tailoring of the interface to the user’s communication needs. In command and control applications, the user most often has the initiative, and in that perspective gestures and speech (the user’s communication channels) have to be carefully studied to support a sensible interaction style. In this chapter, we introduce the notion of semantic frame to integrate gestures and speech in multimodal interfaces. We describe the main elements of a model that has been developed to integrate the use of both channels, and illustrate the model by two fully implemented systems. Possible extensions of the model are presented to improve the supported style, as technologies develop.