Speech and Gestures for Talking Faces in Conversational Dialogue Systems

Abstract

Innovative spoken dialogue systems are beginning to be characterized by designs where interactivity is no longer seen as limited to a series of choices in a question and answer menu approach. New systems strive toward establishing a smooth flow of information modelled on conversational dialogues. In this context, there is currently considerable interest in developing 3D-animated agents to exploit the inherently multimodal nature of speech communication. As the 3D-animation becomes more sophisticated in terms of visual realism, the demand for naturalness in speech and gesture coordination increases. Not only are appropriate and speech-synchronized articulator movements necessary, conversational signals such as cues for turntaking and feedback are also essential. Such conversational signals can be conveyed by both the auditory and visual modality. Verbal (auditory) signals can complement syntax and interact with the prosodic (accentual and phrasal) structure of the utterances. For example, a phrase-final intonation pattern can function as both a cue for prosodic grouping and as a verbal turngiving signal. Gestural (visual) signals such as eyebrow movements and nodding for accentuation can function as parallel signals to intonation (i.e. as linguistic signals) as well as being used as conversational signals (e.g. raised eyebrows to signify an interested, listening agent or nodding to provide encouragement). While much work has been done on describing spoken and gestural conversational signals in human-to-human interaction (see e.g. Allwood, this volume; McNeill, this volume), work aimed at investigating the coordination of these two types of signals in computer-human interaction and the implementation of this knowledge in animated conversational agents is still relatively scarce.