A visual encoding model based on contrastive self‐supervised learning for human brain activity along the ventral visual stream

Abstract

Visual encoding models are important computational models for understanding how information is processed along the visual stream. Many improved visual encoding models have been developed from the perspective of the model architecture and the learning objective, but these are limited to the supervised learning method. From the view of unsupervised learning mechanisms, this paper utilized a pre‐trained neural network to construct a visual encoding model based on con-trastive self‐supervised learning for the ventral visual stream measured by functional magnetic resonance imaging (fMRI). We first extracted features using the ResNet50 model pre‐trained in con-trastive self‐supervised learning (ResNet50‐CSL model), trained a linear regression model for each voxel, and finally calculated the prediction accuracy of different voxels. Compared with the Res‐ Net50 model pre‐trained in a supervised classification task, the ResNet50‐CSL model achieved an equal or even relatively better encoding performance in multiple visual cortical areas. Moreover, the ResNet50‐CSL model performs hierarchical representation of input visual stimuli, which is similar to the human visual cortex in its hierarchical information processing. Our experimental results suggest that the encoding model based on contrastive self‐supervised learning is a strong computational model to compete with supervised models, and contrastive self‐supervised learning proves an effective learning method to extract human brain‐like representations.