Eye movements during free viewing to maximize scene understanding

Abstract

What humans look at and do when freely viewing a scene is not well understood. We measure observer eye movements under different instructions while participants view a customized set of image pairs containing small visual alterations that greatly change scene interpretation (Winograd images). We show that free-viewing fixations resemble those of observers describing scenes but differ from those of observers counting or searching for objects. Fixations are more often directed toward people and objects whose removal most alters scene interpretation, rather than toward the most salient or meaningfully judged regions (meaning maps), or objects perceived to be grasped or gazed at. Small image changes that modify scene understanding (Winograd images), but not salience or meaning maps, alter fixation patterns. By instructing observers to describe scenes while fixating on objects either relevant or irrelevant to scene understanding, we demonstrate that free-viewing eye movements are functionally important for accurate scene comprehension. Thus, an important human default task of free viewing eye movements is to comprehend scenes.