Hearing and seeing beats: The influence of visual beats on the production and perception of prominence

Abstract

Speakers can employ a variety of means to indicate that a word is important, including auditory cues such as pitch accents and visual cues such as manual gestures, head nods and eyebrow movements (visual beats). In this paper, we look at the relation between visual and auditory cues for prominence, based on data collected with an original experimental paradigm in which speakers were instructed to realize a particular target sentence with different distributions of auditory and visual cues. The first experiment revealed that visual beats have a significant effect on the spoken realization of the target words. When a speaker produces a visual beat, the word uttered simultaneously is produced with relatively more spoken emphasis, irrespective of the position of the auditory accent. The second experiment showed that when participants see a speaker realize one of these beat gestures on a word, they perceive this word as more prominent than when they do not see the beat gesture.