AutoLrOpt: An Efficient Optimizer Using Automatic Setting of Learning Rate for Deep Neural Networks

Abstract

In deep learning, various optimizers use a starting value for the learning rate. In general, this value is the one defined by default in the optimizer or chosen by the user after searching for a value giving the best performances of the studied neural network. Finding this value in the case of complex models with large-scale data, either by fine-tuning it manually or by applying a grid search algorithm requires several training sessions, which is penalizing in terms of learning time. To solve this problem, we propose an efficient optimizer using an automatic setting of the learning rate value. Learning then requires a single training session. The proposed optimizer is based on the inexact linear search of Armijo's rule that we adapt for neural networks. We have experimented with this optimizer for several types of loss functions such as mean squared error, mean absolute error, and cross-entropy. Different types of models such as regression and classification are studied. Also, the robustness of this optimizer against the noisy labels is verified. Empirical results demonstrate that the proposed optimizer works well in practice, it outperforms other state-of-the-art optimizers that use default learning rate in terms of accuracy in addition to the time saving when searching for the optimal value of the learning rate.