Novel deep learning model for facial expression recognition based on maximum boosted CNN and LSTM

Abstract

Deep learning has received increasing attention in all fields and has made considerable progress in facial expression recognition (FER). Mainly, the conventional FER methods are trained for constrained datasets, which may not operate well for real-time images. Such real-time image sequences limit the accuracy and efficacy of the traditional system. In this work, the authors present a novel deep learning framework which combines convolutional neural network (CNN) with long short-term memory (LSTM) cell for real-time FER. The novel framework has three main aspects: (i) two different pre-processing techniques are employed to handle illumination variances and to preserve subtle edge information of each image; (ii) correspondingly, the pre-processed images are inputted to the two individual CNN architecture which extracts the spatial features very effectively; (iii) spatial feature maps from two individual CNN layers are fused and integrated with an LSTM layer which extracts temporal relations between the successive frames. They experimented the authors’ proposed method on three publically available FER databases and also with self-created database. With pre-processing, their proposed model achieves comparable and better results to the state-of-the-art.