On the gradient-based sequential tuning of the echo state network reservoir parameters

Abstract

In this paper, the derivative of the input scaling and spectral radius parameters of Echo State Network reservoir are derived. This was achieved by re-writing the reservoir state update equation in terms of template matrices whose eigenvalues can be pre-calculated, so the two parameters appear in the state update equation in the form of simple multiplication which is differentiable. After that the paper derives the derivatives and then discusses why direct application of these two derivatives in gradient descent to optimize reservoirs in a sequential manner would be ineffective due to the nature of the error surface and the problem of large eigenvalue spread on the reservoir state matrix. Finally it is suggested how to apply the derivatives obtained here for joint-optimizing the reservoir and readout at the same time.