Visual and auditory motion information can be used together to
provide complementary information about the movement of objects.
To investigate the neural substrates of such cross-modal integration,
functional magnetic resonance imaging was used to assess brain
activation while subjects performed separate visual and auditory
motion discrimination tasks. Areas of unimodal activation included
the primary and/or early sensory cortex for each modality plus
additional sites extending toward parietal cortex. Areas conjointly
activated by both tasks included lateral parietal cortex, lateral frontal
cortex, anterior midline and anterior insular cortex. The parietal site
encompassed distinct, but partially overlapping, zones of activation
in or near the intraparietal sulcus (IPS). A subsequent task requiring
an explicit cross-modal speed comparison revealed several foci of
enhanced activity relative to the unimodal tasks. These included the
IPS, anterior midline, and anterior insula but not frontal cortex.
During the unimodal auditory motion task, portions of the dorsal
visual motion system showed signals depressed below resting
baseline. Thus, interactions between the two systems involved either
enhancement or suppression depending on the stimuli present and
the nature of the perceptual task. Together, these results identify
human cortical regions involved in polysensory integration and the
attentional selection of cross-modal motion information.
Introduction
A common characteristic of both visual and auditory perception
is the ability to determine the speed and direction of a moving
object, such as an automobile passing on the street. The visual
and auditory sensory information associated with the auto-
mobile presumably merges or becomes coordinated, thereby
producing a unified percept of the movement of the object
within the environment. Additionally, both systems may interact
to coordinate and direct attention to one modality or the other,
and to control subsequent action. However, it remains unclear
how similar the auditory and visual motion systems might be,
and more specifically how and where the two systems interact.
The cortical mechanisms responsible for visual motion
perception have received much study in animals and, more
recently, in humans. In monkeys, the cortical processing of visual
motion is thought to involve a number of anatomically inter-
connected visual areas and their subdivisions referred to, here, as
the dorsal motion pathway. These include lamina 4B in V1, the
thick cytochrome oxidase stripes in V2, areas V3, MT, MST and
possibly lateral and ventral intraparietal areas, LIP and VIP
(Orban et al., 1986; DeYoe and Van Essen, 1988; Desimone and
Ungerleider, 1989; Boussaoud et al., 1990). Information from the
dorsal motion pathway is then thought to inf luence distinct
portions of prefrontal cortex (Wilson et al., 1993; Rao et al.,
1997), presumably for use in directing behavioral responses or
contributing to other cognitive activity.
A similar picture is emerging from neuroimaging and lesion
studies in humans. Areas V1 and V2 in humans are responsive to
visual motion, but more selective responses can be obtained
from extrastriate visual areas located laterally and dorsally in
the occipital and parietal lobes. For instance, hMT, the likely
homolog of the simian middle temporal visual area, MT, is
strongly activated by visual motion stimuli and by tasks involving
a visual motion discrimination (Corbetta et al., 1991; Zeki et al.,
1991; Dupont et al., 1994; Orban et al., 1995; Tootell et al.,
1995a,b; Beauchamp et al., 1997b). Additionally, the same
stimuli and tasks concurrently activate areas in dorsal occipital
cortex and in posterior parietal cortex. Bilateral lesions of lateral
occipital cortex (including hMT) and/or posterior parietal cortex
can selectively compromise visual motion perception, while
leaving auditory and somatosensory motion perception intact
(Zihl et al., 1983, 1991; Rizzo et al., 1995). Together, these
areas may constitute a dorsal motion processing system that is
analogous, if not homologous, to the comparable simian system
(Felleman and Van Essen, 1991).
Compared to our detailed understanding of visual motion
pathways, we know relatively little about pathways for auditory
motion processing. Anatomical studies in monkeys suggest that
there are two auditory streams (as in vision), one of which
includes a system for auditory space analysis that originates in
the caudal belt and parabelt region surrounding primary
auditory cortex and projects to periarcuate cortex (Azuma and
Suzuki, 1984; Romanski et al., 1999). Animal studies of static
sound source localization (Knudsen and Konishi, 1978; Brugge
and Reale, 1985; Phillips and Brugge, 1985; Suga, 1994) have
shown that the location of a sound source can be signaled by
interaural time and/or intensity differences (ITD and IID
respectively). Presumably, some cells can selectively respond to
changes in IID and ITD over time, thereby representing sound
source movement. Indeed, electrophysiological studies in cats
and monkeys have shown that cells selective for auditory motion
exist in primary auditory cortex as well as some subcortical
structures (Sovijärvi and Hyvärinen, 1974; Reale and Brugge,
1990; Ahissar et al., 1992; Stumpf et al., 1992; Takahashi and
Keller, 1992; Toronchuk et al., 1992; Spitzer and Semple, 1993).
However, in primates the identification of a specific system of
interconnected cortical areas for processing auditory motion per
se is currently lacking.
Lesion studies have shown that apparent sound-source
movement in humans can be selectively disrupted when right
parietal and right insular cortex is compromised (Griffiths et
al., 1997). Evidence from human neuroimaging and magneto-
encephalography has shown activation of several cortical
regions by the apparent movement of synthesized sounds,
including the right superior temporal sulcus (STS), primary
auditory and surrounding cortex (PAC+), right insula, right
parietal cortex and right cingulate cortex (Griffiths et al., 1994,
1998; Mäkelä and McEvoy, 1996; Murray et al., 1998; Baumgart
et al., 1999). Despite some inconsistencies across studies, a
picture is emerging of several cortical regions that are activated
A Comparison of Visual and Auditory
Motion Processing in Human Cerebral
Cortex
James W. Lewis, Michael S. Beauchamp
1
and Edgar A. DeYoe
Department of Cell Biology, Neurobiology, and Anatomy,
Medical College of Wisconsin, Milwaukee, WI 53226 and
1
National Institutes of Health, Bethesda, MD, USA
Cerebral Cortex Sep 2000;10:873–888; 1047–3211/00/$4.00 Oxford University Press 2000

during auditory motion processing and may function as a system
for auditory motion analysis.
Where and how the visual and auditory motion systems
interact is not well understood. Such interactions must occur if a
task requires explicit comparison of information from both
modalities. In such instances, information about the direction
and speed of moving objects seems to be derived separately
within each modality, and then compared after conversion to a
common supramodal representation (Stein et al., 1993; Ward
1994; Stein and Wallace, 1996; Driver and Spence, 1998; Snyder
et al., 1998). Presumably, attention is allocated between and
within modalities during such tasks to ensure that the appro-
priate task-relevant information is passed on to decision-making
and behavioral-control systems. Where these various cross-modal
interactions occur in humans is not known.
In monkeys, several cortical areas have been shown to contain
cells that respond to both visual and auditory stimuli, including
temporal cortex (Benevento et al., 1977; Desimone and Gross,
1979; Leinonen et al., 1980; Bruce et al., 1981; Hikosaka et al.,
1988; Watanabe and Iwai, 1991), prefrontal and periarcuate
cortex (Azuma and Suzuki, 1984; Tanila et al., 1992) orbito-
frontal cortex (Benevento et al., 1977) and parietal cortex,
including the lateral intraparietal area, LIP (Mazzoni, 1994;
Linden et al., 1996; Andersen 1997). Anatomical data also
indicate that the ventral intraparietal area (VIP) receives direct
input from both visual- and auditory-related cortex (Lewis and
Van Essen, 2000). However, it is uncertain which of these simian
areas have human homologs and which areas can specifically
contribute to the cross-modal integration of motion information.
Recently, two human imaging studies reported cortical sites
involved with audiovisual integration. Calvert et al. (Calvert et
al., 1999a,b) identified a region in the right superior temporal
sulcus that was more active during integration of aurally and
visually presented language stimuli. Bushara et al. (Bushara et al.,
1999) identified brain areas important for integrating spatial
information across domains in the inferior parietal lobule,
medial frontal cortex and the right inferior temporal cortex.
Suppressive interactions between the auditory and visual
systems have also been noted, though it is unclear whether such
effects are task specific (Haxby et al., 1994; Shulman et al.,
1997) or whether they ref lect uncontrolled cognitive or
attentional factors during the control periods (Shulman et al.,
1997; Binder et al., 1999). The systems responsible for these and
other cross-modal interactions have yet to be fully explored.
In the present study, we used functional magnetic resonance
imaging (fMRI) to examine brain areas subserving visual and
auditory motion processing. Brain activity was examined as
subjects performed separate visual and auditory motion dis-
crimination tasks. We also examined the pattern of activation
when subjects attended to auditory motion, visual motion or
combined audiovisual motion. Because the same subjects
performed both unimodal and cross-modal tasks, we could
distinguish truly convergent cross-modal domains from closely
opposed, but unimodal, domains. The results indicate that visual
and auditory motion processing tasks engage a number of
common cortical regions and pathways that can interact in
different ways depending on the stimuli presented and the
nature of the auditory or visual task. Preliminary reports of these
results have appeared previously (Lewis and DeYoe, 1998a,b).
Materials and Methods
Subjects
Eleven healthy subjects (three females, eight males; age 22–48 years) were
used. Subjects had normal or corrected-to-normal visual acuity and
reported having a normal range of hearing. Ten subjects were strongly
right-handed and one was left-handed. Informed consent was obtained
following guidelines approved by the MCW Human Research Review
Committee.
Isolated Auditory Motion Paradigm
Subjects (n=10) were presented with computer-generated auditory
stimuli (SoundBlaster AWE 64 Gold, Creative Technology Ltd; and Cool
Edit Pro, Syntrillium Software Co.) via electrostatic headphones (Koss
Inc., Milwaukee, WI) that elicited the perception of a moving sound. Each
stimulus consisted of a 300 Hz square wave of duration 500 ms with a
20 ms onset and offset ramp. Interaural intensity differences (IID) elicited
the perception of sound moving through or behind the head from left to
right, with the apparent velocity proportional to the rate of IID change.
Both leftward and rightward motion were randomly presented at one of
Figure 1. Schematic illustration of the auditory and visual motion paradigms. (A) Left depicts the time line of the auditory motion paradigm (224 s total), with a 20 s pre-task baseline
period, and 20 s ON (task) and OFF (control) periods. Middle depicts sound intensity heard in each ear to produce sensation of sound motion based on interaural intensity differences.
Steeper slopes correspond to faster perceived motion. Right inset shows the visual fixation target viewed throughout the entire scan. (B) Left shows the timeline for the isolated visual
motion paradigm. Right illustrates a snapshot of the visual display. Dotted lines indicate bipartite annulus of coherent motion. Refer to Materials and Methods for details.
874 Visual and Auditory Motion Processing ? Lewis et al.