Neural evidence for the flexible control of mental representations

Abstract

This study was designed to explore neural evidence for the simultaneous engagement of multiple mental codes while retaining a visual object in short-term memory (STM) and, if successful, to explore the neural bases of strategic prioritization among these codes. We used multivariate pattern analysis of fMRI data to track patterns of brain activity associated with three common mental codes: visual, verbal, and semantic. When participants did not know which dimension of a sample stimulus would be tested, patterns of brain activity during the memory delay indicated that a visual representation was quickly augmented with both verbal and semantic re-representations of the stimulus. The verbal code emerged as most highly activated, consistent with a canonical visual-to-phonological recoding operation in STM. If participants knew which dimension of a sample stimulus would be tested, brain activity patterns were biased toward the probe-relevant stimulus dimension. Interestingly, probe-irrelevant neural states persisted at an intermediate level of activation when they were potentially relevant later in the trial, but dropped to baseline when cued to be irrelevant. These results reveal the neural dynamics underlying the creation and retention of mental codes, and they illustrate the flexible control that humans can exert over these representations.