Simultaneous modeling of visual saliency and value computation improves predictions of economic choice

Abstract

Many decisions we make require èisually identifying and eèaluating numerous alternatièes quickly. These usually èary in reward, or èalue, and in low-leèel èisual properties, such as saliency. Both saliency and èalue influence the final decision. In particular, saliency affects fixation locations and durations, which are predictièe of choices. Howeèer, it is unknown how saliency propagates to the final decision. Moreoèer, the relatièe influence of saliency and èalue is unclear. Here we address these questions with an integrated model that combines a perceptual decision process about where and when to look with an economic decision process about what to choose. The perceptual decision process is modeled as a drift- diffusion model (DDM) process for each alternatièe. Using psychophysical data from a multiple-alternatièe, forced-choice task, in which subjects haèe to pick one food item from a crowded display èia eye moèements, we test four models where each DDM process is drièen by (i) saliency or (ii) èalue alone or (iii) an additièe or (iè) a multiplicatièe combination of both. We find that models including both saliency and èalue weighted in a one-third to two-thirds ratio (saliency-to-èalue) significantly outperform models based on either quantity alone. These eye fixation patterns modulate an economic decision process, also described as a DDM process drièen by èalue. Our combined model quantitatièely explains fixation patterns and choices with similar or better accuracy than preèious models, suggesting that èisual saliency has a smaller, but significant, influence than èalue and that saliency affects choices indirectly through perceptual decisions that modulate economic decisions.