Processing of changes in visual speech in the human auditory cortex

Abstract

Seeing a talker's articulatory gestures may affect the observer's auditory speech percept. Observing congruent articulatory gestures may enhance the recognition of speech sounds [J. Acoust. Soc. Am. 26 (1954) 212], whereas observing incongruent gestures may change the auditory percept phonetically as occurs in the McGurk effect [Nature 264 (1976) 746]. For example, simultaneous acoustic /ba/ and visual /ga/ are usually heard as /da/. We studied cortical processing of occasional changes in audiovisual and visual speech stimuli with magnetoencephalography. In the audiovisual experiment congruent (acoustic /iti/, visual /iti/) and incongruent (acoustic /ipi/, visual /iti/) audiovisual stimuli, which were both perceived as /iti/, were presented among congruent /ipi/ (acoustic /ipi/, visual /ipi/) stimuli. In the visual experiment only the visual components of these stimuli were presented. A visual change both in audiovisual and visual experiments activated supratemporal auditory cortices bilaterally. The auditory cortex activation to a visual change occurred later in the visual than in the audiovisual experiment, suggesting that interaction between modalities accelerates the detection of visual change in speech. © 2002 Elsevier Science B.V. All rights reserved.