Novel Cognitive Assisted Adaptive Frame Selection for Continuous Sign Language Recognition in Videos Using ConvLSTM

Abstract

People with a hearing impairment commonly use sign language for communication, however, they find it challenging to communicate with a normal person who does not recognise the sign language. They normally require an intermediary human to act as a translator for convenient means of expressing their thoughts. To address this issue, the work aims to enhance their communication capability by eliminating the need for an intermediary person by developing a sign language converter that uses a vision-based dynamic recognition strategy to convert continuous sign language into multimodal output. This work introduces a deep neural network based on convolutional long short-term memory (ConvLSTM) networks to determine the real-time dynamic gesture recognition of the actions of the impaired persons captured through cameras. The investigations of the continuous sign language recognition (CSLR) were deployed on the Chinese Sign Language Dataset, CSL-Daily, Phoenix-2014 and Phoenix-2014T datasets and the performance comparisons were done for conventional LSTM, Gated Recurrent Unit (GRU) and ConvLSTM. Experimental results have shown that the ConvLSTM network outperforms the other techniques, and they can detect the sign actions with a better accuracy of 90%, and a precision rate of 0.93, which ensures interpreting the meanings for each sign sequence with ease by integrating the proposed novel cognitive assisted adaptive keyframe selection. The proposed system could be easily implemented in the modern learning management system.