Review: Sensory Cue Integration

Abstract

In the beginning of the 20th century, there was no Internet and no television. Consequently, the masses would sometimes congregate in a field to try to guess the weight of an ox, in a desperate quest for entertainment. On one of these occasions, Francis Galton collected the guesses made by around 800 participants, and found that their median guess gave a remarkably accurate approximation to the true weight of the ox (Galton 1907). Even though each individual estimate tended to be too high or too low, on average these errors would tend to cancel out. Our brain is in a surprisingly similar situation to that in which Galton found himself at the West of England Fat Stock and Poultry Exhibition in Plymouth. He wished to estimate the true state of affairs of the world (the weight of the ox) in the absence of a direct measurement. Rather, he was relying on a collection of disagreeing, erroneous estimates. In the same way, when we wish to judge the distance to the car in front, the weight of an apple, or the source of a sound, we have only indirect information from our senses upon which to make our judgments. However, by making a number of statistical assumptions, we can show that an appropriate averaging of the information from multiple cues can optimise the precision of our final estimate. This approach of averaging over multiple cues has developed into a rigorous theoretical framework over recent decades, and generated a wealth of empirical data. Trommershäuser, Körding, and Landy's book brings together the wisdom of a crowd of eminent researchers to give an excellent overview of the background, craft, and state-of-the-art of this field. There have been very many recent studies investigating how multiple sources of information might be combined in pursuit of an understanding of a given aspect of our surroundings. The first chapter gives as clear an outline as you could wish for of the approach taken by the vast majority of these. The essential problem is, given multiple noisy, ambiguous, typically mutually contradictory estimates of some property, how can one use all the available information in order to produce the best ensemble estimate? Under a number of assumptions (that the estimates are unbiased, that they are subject to Gaussian noise, and that the best estimator is the one with the minimum variance), then, the optimal approach is rather simple: one weighs the estimates in inverse proportion to their reliability. This idea is developed under the framework of Bayesian decision theory, providing an elegant mathematical framework for what would be optimal. Optimal does not mean 'perfect' but, given the ambiguities and uncertainties inherent in perception, how we could do our best, where our best is defined relative to a particular yardstick [see Bowers and Davis (2012) for a lively discussion of this issue]. The huge advantage of the precision of this approach is that it allows one to determine what we might mean by optimal. The extent to which we achieve this goal, and how this develops through childhood, are addressed with some specific examples in the chapters by Rosas and Wichmann, and Burr, Binda, and Gori. This framework also allows models to encompass the role of prior information (your beliefs in the likely value of the property you are trying to measure before the perceptual information became available). Many of the early chapters in the book address some of the difficulties that emerge when trying to apply the theory. What happens when some of your estimates strongly disagree? How do you decide if two or more sensory cues do indeed come from the same source? In particular, what should you do in a cluttered environment, when there are many potential sources for any particular cue? How do you know the relationship between sensory information and the scene properties that you are interested in, and how do you get to learn what this is? Is human performance really optimal? How do we adapt to a changing environment, or calibrate cues against one another? Simply listing these topics does not, of course, do them justice, but does demonstrate the broad range of difficult ideas covered in the first section of the book. All of these are dealt with in considerable theoretical detail, and this discussion is illuminated by the presentation of relevant empirical data.