ENZO-M — A hybrid approach for optimizing neural networks by evolution and learning

Abstract

ENZO-M combines two successful search techniques using two different timescales: learning (gradient descent) for finetuning of each offspring and evolution for coarse optimization steps of the network topology. Therefore, our evolutionary algorithm is a metaheuristic based on the best available local heuristic. Through training each offspring by fast gradient methods the search space of our evolutionary algorithm is considerably reduced to the set of local optima. Using the parental weights for initializing the weights of each offspring both the gradient descent (learning) is speeded up by 1–2 orders of magnitude and the expected value of the local minimum (fitness of the trained offspring) is far above the mean value for randomly initialized offsprings. Thus, ENZO-M takes full advantage of both the knowledge transfer from the parental genstring using the evolutionary search and the efficiently computable gradient information using finetuning. By the cooperation of the discrete mutation operator and the continuous weight decay method ENZO-M impressively thins out the topology of feedforward neural networks. Especially, ENZO-M also tries to cut off the connections to possibly redundant input units. Therefore ENZO-M not only supports the user in the network design but also recognizes redundant input units.