The eye-ear relationship: investigating auditory impacts on subtitle reading and comprehension

Abstract

The study employs eye-tracking measures and comprehension quizzes to investigate linguistic processing during subtitle reading and comprehension in Arabic and Spanish. Specifically, it examines the impact of native English (L1) audio, foreign Spanish or Arabic (L2) audio, or the absence of audio on subtitle reading strategies and the processing of embedded linguistic units in novice learners. The study utilizes a 2x3 within-subjects design, manipulating audio condition (L1 audio, L2 audio, no audio) and subtitle language (Spanish, Arabic). Global eye-tracking analyses reveal subtitle reading patterns, while local analyses focus on the processing of specific linguistic units. Comprehension quizzes assess learning outcomes after the viewing of the subtitled videos. The findings suggest that the absence of audio leads to increased fixation durations, indicating greater attentional demand. Spanish subtitles result in higher fixation counts and prolonged reading times compared to Arabic, reflecting deeper cognitive processing. The study highlights the complex interplay between audio presence, subtitle language, and individual proficiency in shaping attentional allocation and lexical processing during language acquisition through audio-visual media.