Memory and Attention in Deep Learning

Abstract

This paper has highlighted the advancements made within the deep learning approached through the evolution of attention and memory algorithms. There has been a gradual replacement of the traditional approaches in deep learning through the inclusion of these algorithms that helps in capturing the data based on time and sequence. The model performance is greatly enhanced through the usage of RNNs that uses these algorithms for developing sequential modelling of data. This paper has performed a peer review to understand the different mechanisms that include GRUs, MANN, LSTM and self-attention mechanisms. Memory mechanism assists in capturing the past sequence of datasets for analysing the hidden state within the datasets. Attention mechanisms help in capturing a particular location within a video dataset to understand the patterns of a data. There is an accurate recognition of human actions in the datasets through the implementation of attention mechanisms. This helps in increasing the model performance by enhancing the prediction accuracy and visibility within a particular dataset.