This paper introduces a study about deriving a set of quantitative relationships between speech and co-verbal gestures for improving multimodal input fusion. The initial phase of this study explores the prosodic features of two human communication modalities, speech and gestures, and investigates the nature of their temporal relationships. We have studied a corpus of natural monologues with respect to frequent deictic hand gesture strokes, and their concurrent speech prosody. The prosodic features from the speech signal have been co-analyzed with the visual signal to learn the correlation of the prominent spoken semantic units with the corresponding deictic gesture strokes. Subsequently, the extracted relationships can be used for disambiguating hand movements, correcting speech recognition errors, and improving input fusion for multimodal user interactions with computers. © Springer-Verlag Berlin Heidelberg 2007.
CITATION STYLE
Chen, F., Choi, E. H. C., & Wang, N. (2007). Exploiting speech-gesture correlation in multimodal interaction. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 4552 LNCS, pp. 23–30). Springer Verlag. https://doi.org/10.1007/978-3-540-73110-8_3
Mendeley helps you to discover research relevant for your work.