Computational modeling of multisensory object perception
- ISBN: 9781441956149
- DOI: 10.1007/978-1-4419-5615-6_3
Abstract
Computational modeling largely based on advances in artificial intelligence and machine learning has helped furthering the understanding of some of the principles and mechanisms of multisensory object perception. Furthermore, this theoretical work has led to the development of new experimental paradigms and to important new questions. The last 20 years have seen an increasing emphasis on models that explicitly compute with uncertainties, a crucial aspect of the relation between sensory signals and states of the world. Bayesian models allow for the formulation of such relationships and also of explicit optimality criteria against which human performance can be compared. They therefore allow answering the question, how close human performance comes to a specific formulation of best performance. Maybe even more importantly, Bayesian methods allow comparing quantitatively different models by how well they account for observed data. The success of such techniques in explaining perceptual phenomena has also led to a large number of new open questions, especially about how the brain is able to perform computations that are consistent with these functional models and also about the origin of the algorithms in the brain. We briefly review some key empirical evidence of crossmodal perception and proceed to give an overview of the computational principles evident form this work. The presentation of current modeling approaches to multisensory perception considers Bayesian models, models at an intermediate level, and neural models implementing multimodal computations. Finally, this chapter specifically emphasizes current open questions in theoretical models of multisensory object perception.
Computational modeling of multisensory object perception
Constantin Rothkopf*
Thomas Weisswange
Jochen Triesch
Frankfurt Institute for Advanced Studies (FIAS)
Goethe University Frankfurt
Frankfurt am Main, Germany
*:corresponding author
3.1 Introduction
The brain receives a vast number of sensory signals that relate to a multitude of
meaningful internal representations, reach useful decisions and carry out actions.
Because of the inherently probabilistic nature of all sensing processes one of their
fundamental properties is their associated uncertainty. Sources of uncertainty include
the neural noise due to physical processes of transduction in early stages of neural
encoding, noise due to physical constraints such as unavoidable aberrations of every
imaging device, and uncertainties because many environmental states can give rise
to the same sensory measurement as well as many different sensory measurements
can be evoked by the same state of the world. All these uncertainties render the
inverse computation from sensory signals to states of the world difficult, but the brain
gives humans the perception of a stable and mostly unambiguous world. Vision,
audition, touch, proprioception and all other senses suggest to us a world of
individual objects and well defined states. How can the brain do this?
Over the last decades ample data has been assembled in order to shed light on how
the human and primate brains are able to accomplish this feat. The literature on
empirical investigations of cue combination, cue integration, perception as an
inference process, and many related aspects is vast (see e.g. Kersten, Mamassian,
Yuille, 2004; Ernst, Bülthoff, 2004; Yuille, Kersten, 2006 for reviews).
Psychophysical, neurophysiological, and imaging studies have quantified human and
primate performance and knowledge on the neural implementation has accumulated.
Nevertheless, the question on how the brain merges sensory inputs into complete
percepts offers many unsolved problems. Sensory processing has been traditionally
thought to be separate in the respective modalities. The segregation has been
applied even within modalities, as e.g. in the separation of ventral and dorsal streams
in vision, promoting a hierarchical and modular view of sensory processing. But
recent experimental results have emphasized the multimodal processing of sensory
stimuli even in areas previously regarded as unimodal.
Theoretical work largely based on advances in artificial intelligence and machine
learning has not only furthered the understanding of some of the principles and
mechanisms of this process but also led to important questions and new
experimental paradigms. The last 20 years have seen an increasing emphasis on
Bayesian techniques in multimodal perception, mostly because such models
explicitly represent uncertainties, a crucial aspect of the relation between sensory
signals and states of the world. Bayesian models allow for the formulation of such
relationships and also of explicit optimality criteria against which human performance
can be compared. They therefore allow answering the question, how close human
performance comes to a specific formulation of best performance. Maybe even more
importantly, Bayesian methods allow comparing quantitatively different models by
how well they account for observed data.
The success of Bayesian techniques in explaining a large variety of perceptual
phenomena has also led to a large number of additional open questions, especially
about how the brain is able to perform computations that are consistent with the
functional models and also about the origin of these models. For this reason, and
because comprehensive review articles of perceptual processes and their modeling
have been published in the past years (see e.g. Kersten, Mamassian, Yuille, 2004;
Knill, Pouget, 2004; Ernst, Bülthoff, 2004; Yuille, Kersten, 2006; Shams & Seitz,
2008), this book chapter specifically emphasizes open questions in theoretical
models of multisensory object perception.
3.2 Empirical evidence for Crossmodal Object Perception
Sign up today - FREE
Mendeley saves you time finding and organizing research. Learn more
- All your research in one place
- Add and import papers easily
- Access it anywhere, anytime
