Optimization methods for deep neural networks

Abstract

This paper reviews the optimization methods for the training of deep neural networks, particularly with complex problems. The primary goal of the optimizer is to speed up the training and helps to boost the efficiency of the model. Optimization methods are the engines underlying deep neural networks that enable them to learn from data. This review paper covers the fundamentals of gradient-based optimization methods and their application to training neural networks. The major topics reviewed in this paper are gradient descent, momentum methods, 2nd order methods, and stochastic methods. The strengths of the optimization methods and their shortcomings, potential research directions, and open topics are being addressed and highlighted.