Visual Speech Reduces Cognitive Effort as Measured by EEG Theta Power and Pupil Dilation

Abstract

Listening effort reflects the cognitive and motivational resources allocated to speech comprehension, particularly under challenging conditions. Visual cues are known to enhance speech perception, potentially by reducing the cognitive demands of the task. However, the neurophysiological mechanisms underlying this facilitation, especially in terms of effort-related changes, remain unclear. In this study, we combined pupillometry and electroencephalography (EEG) to investigate how visual speech cues modulate cognitive effort during speech recognition. Twenty-two participants (seven females) performed a speech-in-noise task under three modalities: (1) auditory-only, (2) audiovisual, and (3) visual-only. Task difficulty was manipulated via signal-to-noise ratio (SNR) in the first two modalities. Firstly, we found an inverted U-shape relationship between pupil dilation and frontal midline theta with SNR for audiovisual and auditory-only speech, consistent with prior models of effort allocation. Secondly, we observed the SNR at which the neurophysiological measures peaked was at a lower SNR for audiovisual speech. Surprisingly, we found pupil dilation to be larger overall in audiovisual speech, while frontal midline theta did not show differences in either modality. These findings highlight the complexity of interpreting physiological markers of effort and suggest that visual cues may alter the temporal dynamics or resource allocation strategies during speech processing. Our results support the extension of auditory-based models of listening effort to audiovisual contexts and underscore the value of integrating multimodal neurophysiological measures to better understand the cognitive and neural mechanisms of effortful listening.