Soft committee machine using simple derivative term

Abstract

In on-line gradient descent learning, the local property of the derivative of the output function can cause slow convergence. This phenomenon, called a plateau, occurs in the learning process of the multilayer network. Improving the derivative term, we employ the proposed method replacing the derivative term with a constant that greatly increases the relaxation speed. Moreover, we replace the derivative term with the 2nd order of expansion of the derivative, and it beaks a plateau faster than the original method. © 2014 Springer International Publishing.