ARTICLE Communicated by James Williamson
Neural Systems as Nonlinear Filters
Wolfgang MaassInstitute for Theoretical Computer Science, Technische Universita¨t Graz, A-8010 Graz,AustriaEduardo D. SontagDepartment of Mathematics, Rutgers University, New Brunswick, NJ 08903, U.S.A.
Experimental data show that biological synapses behave quite differentlyfrom the symbolic synapses in all common artiŽcial neural network mod-els. Biological synapses are dynamic; their “weight” changes on a shorttimescale by several hundred percent in dependence of the past inputto the synapse. In this article we address the question how this inherentsynaptic dynamics (which should not be confused with long term learn-ing) affects the computational power of a neural network. In particular,we analyze computations on temporal and spatiotemporal patterns, andwe give a complete mathematical characterization of all Žlters that can beapproximated by feedforward neural networks with dynamic synapses.It turns out that even with just a single hidden layer, such networks canapproximate a very rich class of nonlinear Žlters: all Žlters that can becharacterized by Volterra series. This result is robust with regard to var-ious changes in the model for synaptic dynamics. Our characterizationresult provides for all nonlinear Žlters that are approximable by Volterraseries a new complexity hierarchy related to the cost of implementingsuch Žlters in neural systems.1 IntroductionSynapses in common artiŽcial neural network models are static: the valuewi of a synaptic weight is assumed to change only during “learning.” Incontrast to that, the “weight” wi (t) of a biological synapse at time t is knownto be strongly dependent on the inputs xi(t¡t ) that this synapsehas receivedfrom the presynaptic neuron i at previous time steps t¡t . Varela et al. (1997)have shown that a model of the formwi(t) D wi ¢ D(t) ¢ (1 C F(t)) (1.1)with a constant wi, a depression term D(t) with values in (0, 1], and a facil-itation term F(t) ¸ 0 can be Žtted remarkably well to experimental data forsynaptic dynamics. The facilitation term F(t) is usually modeled as a linear
Neural Computation 12, 1743–1772 (2000) c° 2000 Massachusetts Institute of Technology

1744 Wolfgang Maass and Eduardo D. Sontag
Žlter with exponential decay: If xi(t ¡ t ) is the output of the presynapticneuron (typically modeled by a sum of d-functions), then the current valueof this facilitation term is of the formF(t) D r Z 10 xi(t ¡ t ) ¢ e¡t /c dt (1.2)for certain parameters r , c > 0 that vary from synapse to synapse. A fewother models have been proposed for synaptic dynamics (see e.g. Dobrunz& Stevens, 1997; Murthy, Sejnowski, & Stevens, 1997; Tsodyks, Pawelzik, &Markram, 1998; Koch, 1999; Maass & Zador, 1998, 1999) that are all quitesimilar. Closely related models had already been proposed and investigatedin Grossberg (1969, 1972, 1984); Francis, Grossberg, & Mingolla, 1994). Ouranalysis in this article is primarily based on the model of Varela et al. (1997).However we will prove that our results also hold for the somewhat morecomplex model for synaptic dynamics in a mean-Želd context of Tsodyks etal. (1998).We show that such inherent synaptic dynamics empower neural net-works with a remarkable capability for carrying out computations on tem-poral patterns (i.e., time series) and spatiotemporal patterns. This compu-tational mode, where inputs and outputs consist of temporal patterns orspatiotemporal patterns—rather than static vectors of numbers—appearsto provide a more adequate framework for analyzing computations in bio-logical neural systems. Furthermore their capability for processing tempo-ral and spatiotemporal patterns in a very efŽcient manner may be linked totheir superior capabilities for real-time processing of sensory input; hence,our analysis may provide new ideas for designing artiŽcial neural systemswith similar capabilities.We consider not just computations ofneural systemswith a single tempo-ral pattern as input, but also characterize their computational power for thecase where several different temporal patterns u1(t), . . . , un(t) are presentedin parallel as input to the neural system. Hence we also provide a completecharacterization of the computational power of feedforward neural systemsfor the case where salient information is encoded in temporal correlationsof Žring activity in different pools of neurons (represented by correlationsamong the corresponding continuous functions u1(t), . . . , un(t)). Therefore,various informal suggestions for computational uses of such code can beplaced on a rigorous mathematical foundation. It is easy to see that a largevariety of computational operations that respond in a particular mannerto correlations in temporal input patterns deŽne time-invariant Žlters withfading memory; hence they can in principle be implemented on each of thevarious kinds of dynamic networks considered in this article.Previous standard models for computations on temporal patterns in ar-tiŽcial neural networks are time-delay neural networks (where temporalstructure is transformed into spatial structure) and recurrent neural net-works, both being based on standard “static” synapses (Hertz, Krogh, &