Recurrent neural network with dynamic memory

Abstract

Recurrent neural network presents great performance and learning capability in dealing with sensory processing, sequence learning and reinforcement learning. However, the memory ability of the RNNs is limited because of the gradient vanishing problem, the memory conflict and the very limited memory capability. In order to overcome these defects, we propose a novel RNN model, which is called RNN-DM. The proposed model has two types of memory: internal memory and external memory. The internal memory exists in the neuron unit of hidden layer, which is used to solve the memory conflict problem and reduce the influence of gradient vanishing problem during network learning. The external memory is a memory matrix, which is used to store the complex data structure and variable, it has ability to solve the limited memory capacity problem.