A Multi-Fidelity Genetic Algorithm for Hyperparameter Optimization of Deep Neural Networks

Abstract

Hyperparameter optimization on machine learning models is crucial for their correct refinement. For complex big models such as deep learning (DL) models, in which a single training model is supposed to have a very high computational cost, this optimization sometimes becomes unfeasible. Multi-fidelity optimization algorithms are a solution to alleviate this computational cost of optimizing hyperparameters of DL models. In this scope, we propose genetic algorithm based on multi-fidelity evaluations with 2 objective, a new multi-fidelity algorithm that relies on a genetic algorithm. This article clearly defines how to adapt the evolutionary scenario to follow the multi-fidelity approach, and we propose a new scheme to evaluate each individual based on the use of two objectives: the result of the low-fidelity evaluation and the learning capacity, with the use of the latter being novel during the evaluation process. Our experimental section allows us to show how our proposal improves the state-of-the-art in different classification and regression problems.