Adaptive self-scaling non-monotone BFGS training algorithm for recurrent neural networks

Abstract

In this paper, we propose an adaptive BFGS, which uses a self-adaptive scaling factor for the Hessian matrix and is equipped with nonmonotone strategy. Our experimental evaluation using different recurrent networks architectures provides evidence that the proposed approach trains successfully recurrent networks of various architectures, inheriting the benefits of the BFGS and, at the same time, alleviating some of its limitations. © Springer-Verlag Berlin Heidelberg 2007.