A multimodal fission approach with a presentation agent in the dialog system smartKom

Abstract

One of the major scientific goals of SmartKom is to design a new human-machine interaction metaphor for a multimodal dialog system that combines speech, gesture, and mimics input with speech, gesture, mimics and graphics output. In SmartKom an animated life-like character, serves as the communication partner of the user. In this paper we focus on the multimodal output of SmartKom, performed by MMoPaD (Multi Modal Presentation and Display), showing how these highly ambitious tasks of the SmartKom system are managed and realized on the output side, i.e., how the communication assistant adapts its behavior to the available output media and modalities with respect to the current dialog situation. Depending on them presentations are split in parts (fission), which are output in different ways. An important task is the synchronization of gestures, mimics and speech of the presentation agent Smartakus which enriches the presentation and makes it more lively. The system is used on standard and mobile devices.