Fast Spectral Radius Initialization for Recurrent Neural Networks

Abstract

In this paper we address the problem of grounded weights initialization for Recurrent Neural Networks. Specifically, we propose a method, rooted in the field of Random Matrix theory, to perform a fast initialization of recurrent weight matrices that meet specific constraints on their spectral radius. Focusing on the Reservoir Computing (RC) framework, the proposed approach allows us to overcome the typical computational bottleneck related to the eigendecomposition of large matrices, enabling to efficiently design large reservoir networks and hence to address time-series tasks characterized by medium/big datasets. Experimental results show that the proposed method enables an accurate control of the spectral radius of randomly initialized recurrent matrices, providing an initialization approach that is extremely more efficient compared to common RC practice.