Investigating Lexical and Semantic Cognition by Using Neural Network to Encode and Decode Brain Imaging

Abstract

The question of how the human brain represents conceptual knowledge has received significant attention in many scientific fields. Over the last decade, there has been increasing interest in the use of deep learning methods for analyzing functional magnetic resonance imaging (fMRI) data. In this paper, we report a series of experiments with neural networks for fMRI encoding and decoding. Results show that by using neural networks, both encoding and decoding accuracies are improved compared to a linear model on the same input. To evaluate the contextual information influences in cognitive modeling, we also extend the stimuli dataset from single noun to description sentences. The experiments indicate the impact of context information varies from person to person. To illustrate the strong correlation between linguistic and visual representations in the human brain, we extend the stimuli from a single word to images which were not present to the participant during fMRI data collection.