Coordination in Sensory Integration
Jochen Triesch, Constantin Rothkopf, Thomas Weisswange
Frankfurt Institute for Advanced Studies
Ruth-Moufang-Str. 1, 60437 Frankfurt a.M., Germany
Introduction
Coordination often refers to "the harmonious functioning of parts for effective results"1 and as such it
is omnipresent in brain and behavior. During perception, the brain has to make sense of sensory
signals from a number of different modalities such as vision, audition, olfaction, touch, proprioception,
etc. These signals need to be processed and integrated in order to compute a (usually correct)
interpretation of the environment. How this is happening is a fundamental problem. In the following,
we will raise a number of central questions regarding sensory integration paying special attention to
dynamic coordination in the brain. A more detailed review of sensory integration is provided
elsewhere (Rothkopf, Weisswange and Triesch, in press).
Why integrate?
Perception is a difficult computational problem. The state of the world must be inferred from noisy
and ambiguous sensory signals. The brain's solution relies on making use of all available sources of
information. This refers to the different sensory modalities mentioned above (e.g., Stein & Meredith,
1993), but also to different so-called cues within one modality. For example, there are many so-called
depth cues in visual perception, that are thought to contribute to the perception of an object's
distance from the observer (e.g., Landy, Maloney, Johnston and Young, 1995). Next to the
integration of various sources of evidence, perception also heavily utilizes previously acquired
knowledge about the world. Such prior information will be particularly important when the sensory
data are very ambiguous (e.g., Weiss, Simoncelli and Adelson, 2002) and it may be either innate or
the result of learning and subject to constant adaptation.
The benefit of combining several sources of information is twofold. First, our estimates of the state of
the world will become more accurate as we integrate several noisy sources of information. The lion
share of work on sensory integration has focused on this aspect and there are many demonstrations
of this in humans and various animal species. Second, we may be able to respond more quickly, i.e.
processing time is reduced when stimuli are presented in more than one modality. Both aspects are
of obvious relevance for an organism's survival and well-being and can be closely related: when
several sources of noisy evidence are available, then, under certain assumptions, we can obtain the
same amount of information from them if we observe a single source for a long time or several of
them for a correspondingly shorter time.
How to integrate?
A natural starting point for asking how the brain might integrate sensory information from different
cues or modalities is the question what the optimal solution to the problem is. Such questions can be
answered in the popular framework of Bayesian inference (Pearl, 1988), for which many good
reviews are available (Kersten, Mamassian and Yuille ,2004; Kersten & Yuille, 2003; Yuille & Kersten,
2006). In this framework one can construct so-called "ideal observers" that use all the available
sensory information in an optimal fashion according to the laws of probability and statistics. After the
ideal observer has been constructed and its behavior has been analyzed, it can be compared to that
1 Source: http://www.merriam-webster.com/

of human subjects or animals. In many (simple) situations, it has been shown that human behavior is
well-modeled by an appropriate ideal observer model. This is usually taken as evidence that the brain
performs Bayesian inference.
Unfortunately, however, solving the Bayesian inference problem and constructing appropriate ideal
observer models can be a very difficult task. In the most general setting, Bayesian inference belongs
to a class of computational problems that requires an exponentially increasing amount of processing
as the problem size gets bigger, e.g. the more sensory variables are involved. In these situations, it
may be infeasible to construct the ideal observer and approximations have to be made. As a
consequence, it cannot be judged whether human behavior is optimal either. However, since the
brain will also have to use approximations to solve the Bayesian inference problem, it is important to
ask what kinds of approximations it is using. In principle, this question can be answered within the
Bayesian inference framework, but we do not know of specific examples, where this has been
demonstrated.
Finally, viewing cue integration (and more generally perception) as Bayesian inference is a
description entirely at the computational level. It is still unclear how the neural implementation of
Bayesian inference (or approximations of it) could look like at the level of groups of neurons
exchanging action potentials. This is in fact one of the most important and pressing questions in the
field of Computational Neuroscience (Ma, Beck, Latham and Pouget, 2006; Deneve, 2005).
How does the brain learn how to integrate?
The Bayesian inference framework is a powerful framework for studying sensory integration. But how
does the brain acquire the necessary probabilistic models? How does it decide what sensory
variables to represent? How does it learn their statistical relationships? The machine learning and
statistics communities have made some progress regarding how such models and their parameters
can be learned, but optimal Bayesian learners are even harder to construct than ideal observers and
human learning can deviate strongly from the ideal case.
Experimental evidence regarding the acquisition of sensory integration abilities comes from
developmental studies with children and learning experiments with adults. Interestingly, recent
experiments with children suggest that it may take many years before children exhibit appropriate
sensory integration abilities consistent with ideal observer models (Nardini, Jones, Bedford and
Braddick, 2008; Gori, Del Viva, Sandini and Burr, 2008; Neil, Chee-Ruiter, Scheier, Lewkowicz,
Shimojo, 2006). Initially, they may not be integrating different modalities at all (Gori et al., 2008).
Regarding adult learning experiments, a relatively simple case is the one where the set of different
cues is fixed and only their relative weighting is changing. In visual cue integration, for example,
Ernst, Banks and Bülthoff (2000) and Atkins, Fiser and Jacobs (2001) showed that when two
conflicting visual cues are paired with a haptic cue, subjects will, over the course of a few days, learn
to increase the weight of the visual cue that is consistent with the haptic cue and decrease the weight
of the inconsistent cue. Thus, it appears that the haptic cue is serving as a reference model for
adjusting the visual cues.
When to integrate?
Another question of fundamental importance is when signals from different modalities or cues should
be combined or when they should be considered separately (Koerding, Beierholm, Ma, Quartz,
Tenenbaum and Shams, 2007). An interesting problem in this context is that of audiovisual source
localization. Imagine your task is to estimate the location of one or two target objects that are
presented simultaneously in the auditory and/or visual domain through brief light flashes and sounds.