J Comput Neurosci (2011) 30:1–5
DOI 10.1007/s10827-011-0314-3
Information theory in neuroscience
Alexander G. Dimitrov · Aurel A. Lazar ·
Jonathan D. Victor
Published online: 29 January 2011
© Springer Science+Business Media, LLC 2011
Information Theory started and, according to some,
ended with Shannon’s seminal paper “A Mathematical
Theory of Communication” (Shannon 1948). Because its
significance and flexibility were quickly recognized, there
were numerous attempts to apply it to diverse fields
outside of its original scope. This prompted Shannon
to write his famous essay “The Bandwagon” (Shannon
1956), warning against indiscriminate use of the new
tool. Nevertheless, non-standard applications of Infor-
mation Theory persisted.
Very soon after Shannon’s initial publication (Shannon
1948), several manuscripts provided the foundations
of much of the current use of information theory
in neuroscience. MacKay and McCulloch (1952) ap-
plied the concept of information to propose limits of
the transmission capacity of a nerve cell. This work
A. G. Dimitrov (B)
Department of Mathematics and Science Programs,
Washington State University Vancouver,
14204 NE Salmon Creek Ave,
Vancouver, WA 98686, USA
e-mail: alex.dimitrov@vancouver.wsu.edu
A. A. Lazar
Department of Electrical Engineering, Columbia University,
Mail Code 4712,
500 West 120th Street,
New York, NY 10027, USA
e-mail: aurel@ee.columbia.edu
J. D. Victor
Division of Systems Neurology and Neuroscience,
Department of Neurology and Neuroscience,
Weill Cornell Medical College,
1300 York Avenue,
New York, NY 10065, USA
e-mail: jdvicto@med.cornell.edu
foreshadowed future work on what can be termed
“Neural Information Flow”—how much information
moves through the nervous system, and the constraints
that information theory imposes on the capabilities of
neural systems for communication, computation and
behavior. A second set of manuscripts, by Attneave
(1954) and Barlow (1961), discussed information as
a constraint on neural system structure and function,
proposing that neural structure in sensory system is
matched to statistical structure of the sensory environ-
ment, in a way to optimize information transmission.
This is the main idea behind the “Structure from Infor-
mation” line of research that is still very active today. A
third thread, “Reliable Computation with Noisy/Faulty
Elements”, started both in the information-theoretic
community (Shannon and McCarthy 1956) and neuro-
science (Winograd and Cowan 1963). With the advent
of integrated circuits that were essentially faultless,
interest began to wane. However, as IC technology
continues to push towards smaller and faster computa-
tional elements (even at the expense of reliability), and
as neuromorphic systems are developed with variability
designed in (Merolla and Boahen 2006), this topic is
gaining in popularity again in the electronics commu-
nity, and neuroscientists again may have something to
contribute to the discussion.
1 Subsequent developments
The theme that arguably has had the widest influence
on the neuroscience community, and is most heavily
represented in the current special issue of JCNS, is
that of “Neural Information Flow”. The initial works of
MacKay and McCulloch (1952), McCulloch (1952) and

2 J Comput Neurosci (2011) 30:1–5
Rapoport and Horvath (1960) showed that neurons are
in principle able to relay large quantities of informa-
tion. This research lead to the first attempts to charac-
terize the information flow in specific neural systems
(Werner and Mountcastle 1965), and also started the
first major controversy in the field, which still resonates
today: the debate about timing versus frequency codes
(Stein 1967; Stein et al. 1972). A steady stream of
articles followed, both discussing these hypothesis and
attempting to clarify the type of information relayed by
nerve cells (Abeles and Lass 1975; Eagles and Purple
1974; Eckhorn and Pöpel 1974; Eckhorn et al. 1976;
Harvey 1978; Lass and Abeles 1975; Norwich 1977;
Poussart 1971; Stark et al. 1969; Taylor 1975; Walloe
1970).
After the initial rise in interest, the application of
Information Theory to neuroscience was extended to
a few more systems and questions (Eckhorn and Pöpel
1981; Eckhorn and Querfurth 1985; Fuller and Williams
1983; Kjaer et al. 1994; Lestienne and Strehler 1987,
1988; Optican and Richmond 1987; Surmeier and
Weinberg 1985; Tsukuda et al. 1984; Victor and
Johanessma 1986), but did not spread too broadly. This
was presumably because, despite strong theoretical
advances in Information Theory, its applicability was
hampered by difficulty in measuring and interpreting
information-theoretic quantities.
The work of de Ruyter van Steveninck and Bialek
(1988) started what could be called the modern era
of information-theoretic analysis in neuroscience, in
which Information Theory is seeing more and more
refined applications. Their work advanced the concep-
tual aspects of the application of information theory to
neuroscience and, subsequently, provided a relatively
straightforward way to estimate information-theoretic
quantities (Strong et al. 1998). This work provided an
approach to removing biases in information estimates
due to finite sample size, but the scope of applicability
of their approach was limited. The difficulties in obtain-
ing unbiased estimates of information-theoretic quanti-
ties were noted early on by Carlton (1969) and Miller
(1955) and brought again to attention by Treves and
Panzeri (1995). However, it took the renewed interest
generated by Strong et al. (1998) to spur the research
that eventually resolved them. Almost simultaneously,
several groups provided the neuroscience community
with robust estimators valid under different conditions
(Kennel et al. 2005; Nemenman et al. 2004; Paninski
2003; Victor 2002). This diversity proved important, as
Paninski (2003) proved an inconsistency theorem show-
ing that most common estimation techniques can
encounter conditions that lead to arbitrary poor
estimates.
At the same time, several other branches of Infor-
mation Theory saw application in neuroscience context.
The introduction of techniques stemming from work on
quantization and lossy compression (Gersho and Gray
1991) provided lower bounds of information-theoretic
quantities and ideas about inference based on them
(Dimitrov and Miller 2001; Samengo 2002; Tishby et al.
1999). Furthermore, a large class of neuron models
were characterized as samplers that, under appropriate
conditions, faithfully encode sensory information in the
spike train (time encoding machines, Lazar 2004). The
class of neurons includes integrate-and-fire (Lazar and
Pnevmatikakis 2008), threshold-and-fire (Lazar et al.
2010) and Hodgkin-Huxley neurons (Lazar 2010).
2 Current state
The work presented in the current issue builds on the
developments in both information-theoretic and exper-
imental techniques.
Several of the contributions apply a recent develop-
ments in Information Theory - directed information—
to clarifying the structure of biological neural networks
from observations of their activity. The works origi-
nate from the ideas of Granger (1969) on causal inter-
actions, and was placed in an information-theoretic
perspective by Massey (1990), Massey and Massey (2005)
and Schreiber (2000). Amblard and Michel (2011)
here merge the two ideas and extract Granger causal-
ity graphs by using directed information measures.
They show that such tools are needed to analyze the
structure of systems with feedback in general, and
neural systems specifically. The authors also provide
practical approximations with which to estimate these
structures. Quinn et al. (2011) present a novel, non-
linear robust extension of the linear Granger tools
and use it to infer the dynamics of neural ensembles
based on physiological observations. In particular, the
procedure uses point process models of neural spike
trains, performs parameter and model order selec-
tion with minimal description length, and is applied
to the analysis of interactions in neuronal assemblies
in the primary motor cortex (MI) of macaque mon-
keys. Vicente et al. (2011) investigate transfer entropy
(TE) as an alternative measure of effective connectiv-
ity to electrophysiological data based on simulations
and magnetoencephalography (MEG) recordings in a
simple motor task. The authors demonstrate that TE
improved the detectability of effective connectivity for
non-linear interactions, and for sensor level MEG sig-
nals where linear methods are hampered by signal-
cross-talk due to volume conduction. Using neocortical