Neocortical and Hippocampal Theta Oscillations Track Audiovisual Integration and Replay of Speech Memories

Abstract

“Are you talkin’ to me?!” If you ever watched the masterpiece “Taxi Driver” directed by Martin Scorsese, you certainly recall the monologue during which Travis Bickle rehearses an imaginary confrontation in front of a mirror. While remembering this scene, you recollect a myriad of speech features across visual and auditory senses with a smooth sensation of unified memory. The aim of this study was to investigate how the fine-grained synchrony between coinciding visual and auditory features impacts brain oscillations when forming multisensory speech memories. We developed a memory task presenting participants with short synchronous or asynchronous movie clips focused on the face of speakers in real interviews, all the while undergoing magnetoencephalography recording. In the synchronous condition, the natural alignment between visual and auditory onsets was kept intact. In the asynchronous condition, auditory onsets were delayed to present lip movements and speech sounds in antiphase specifically with respect to the theta oscillation synchronizing them in the original movie. Our results first showed that theta oscillations in the neocortex and hippocampus were modulated by the level of synchrony between lip movements and syllables during audiovisual speech perception. Second, theta asynchrony between the lip movements and auditory envelope during audiovisual speech perception reduced the accuracy of subsequent theta oscillation reinstatement during memory recollection. We conclude that neural theta oscillations play a pivotal role in both audiovisual integration and memory replay of speech.