Modular Neural Networks

Abstract

In the previous chapters we have discussed different models of neural networks – linear, recurrent, supervised, unsupervised, self-organizing, etc. Each kind of network relies on a different theoretical or practical approach. In this chapter we investigate how those different models can be combined. We transform each single network in a module that can be freely intermixed with modules of other types. In this way we arrive at the concept of modular neural networks. Several general issues have led to the development of modular systems [153]: • Reducing model complexity. As we discussed in Chap. 10 the way to reduce training time is to control the degrees of freedom of the system. • Incorporating knowledge. Complete modules are an extension of the ap-proach discussed in Sect. 10.3.5 of learning with hints. • Data fusion and prediction averaging. Committees of networks can be con-sidered as composite systems made of similar elements (Sect. 9.1.6) • Combination of techniques. More than one method or class of network can be used as building block. • Learning different tasks simultaneously. Trained modules may be shared among systems designed for different tasks. • Robustness and incrementality. The combined network can grow gradually and can be made fault-tolerant. Modular neural networks, as combined structures, have also a biological back-ground: Natural neural systems are composed of a hierarchy of networks built of elements specialized for different tasks. In general, combined networks are more powerful than flat unstructured ones. 16.1 Constructive algorithms for modular networks Before considering networks with a self-organizing layer, we review some tech-niques that have been proposed to provide structure to the hidden layer of R. Rojas: Neural Networks, Springer-Verlag, Berlin, 1996 414 16 Modular Neural Networks feed-forward neural networks or to the complete system (as a kind of decision tree). 16.1.1 Cascade correlation An important but difficult problem in neural network modeling, is the se-lection of the appropriate number of hidden units. The cascade correlation algorithm, proposed by Fahlman and Lebiere [131], addresses this issue by recruiting new units according to the residual approximation error. The algo-rithm succeeds in giving structure to the network and reducing the training time necessary for a given task [391]. Figure 16.1 shows a network trained and structured using cascade cor-relation. Assume for the moment that a single output unit is required. The algorithms starts with zero hidden units and adds one at a time according to the residual error. The diagram on the right in Figure 16.1 shows the start configuration. The output unit is trained to minimize the quadratic error. Training stops when the error has leveled off. If the average quadratic error is still greater than the desired upper bound, we must add a hidden unit and retrain the network. output