Multisensory naturalistic decoding with high-density diffuse optical tomography

Abstract

SIGNIFICANCE Decoding naturalistic content from brain activity has important neuroscience and clinical implications. Information about visual scenes and intelligible speech has been decoded from cortical activity using functional magnetic resonance imaging (fMRI) and electrocorticography, but widespread applications are limited by the logistics of these technologies. AIM High-density diffuse optical tomography (HD-DOT) offers image quality approaching that of fMRI but with the silent, open scanning environment afforded by optical methods, thus opening the door to more naturalistic research and applications. Although early visual decoding studies with HD-DOT have been promising, decoding of naturalistic auditory and multisensory stimulus information from HD-DOT data has not been established. APPROACH Audiovisual decoding was investigated using HD-DOT data collected from participants who viewed a library of movie clips. A template-matching strategy was used to decode which movie clip a participant viewed based on their HD-DOT data. Factors affecting decoding performance-including trial duration and number of decoding choices-were systematically evaluated. RESULTS Decoding accuracy was 94.2% for four-way decoding utilizing 4 min of data per trial as a starting point. As parameters were made more stringent, decoding performance remained significantly above chance with strong effect sizes down to 15-s trials and up to 32 choices. Comparable decoding accuracies were obtained when cortical sampling was confined to visual and auditory regions and when participants were presented with purely auditory or visual clips. CONCLUSIONS HD-DOT data sample cortical hemodynamics with sufficient resolution and fidelity to support decoding complex, naturalistic, multisensory stimuli via template matching. These results provide a foundation for future studies on more intricate decoding algorithms to reconstruct diverse features of novel naturalistic stimuli from HD-DOT data.