On-Line Learning in Multilayer Neural Networks

Abstract

We present an analytic solution to the problem of on-line gradient-descent learning for two-layer neural networks with an arbitrary number of hidden units in both teacher and student networks. The technique, demonstrated here for the case of adaptive input-to-hidden weights, becomes exact as the dimensionality of the input space increases. Layered neural networks are of interest for their ability to implement input-output maps 1]. Classiication and regression tasks formulated as a map from an N-dimensional input space onto a scalar are realized through a map = f J (), which can be modiied through changes in the internal parameters fJg specifying the strength of the interneuron couplings. Learning refers to the modiication of these couplings so as to bring the map f J implemented by the network as close as possible to a desired map ~ f. Information about the desired map is provided through independent examples (; ;), with = ~ f() for all. A recently introduced approach investigates on-line learning 2]. In this scenario the couplings are adjusted to minimize the error after the presentation of each example. The resulting changes in fJg are described as a dynamical evolution, with the number of examples playing the role of time. The average that accounts for the disorder introduced by the independent random selection of an example at each time step can be performed directly. The result is expressed in the form of dynamical equations for order parameters which describe correlations among the various nodes in the trained network as well as their degree of specialization towards the implementation of the desired task. Here we obtain analytic equations of motion for the order parameters in a general two-layer scenario: a student network composed of N input units, K hidden units, and a single linear output unit is trained to perform a task deened through a teacher network of similar architecture except that its number M of hidden units is not necessarily equal to K. Two-layer networks with an arbitrary number of hidden units