Application of deep learning in Mandarin Chinese lip-reading recognition

Abstract

Lip-reading is an emerging technology in recent years, and it can be applied to the field of language recovery, criminal investigation, identity authentication, etc. We aim to recognize what the speaker is saying without audio but only video. Because of the different mouth shapes and the influence of homophones, the current Mandarin Chinese lip-reading network is proposed, an end-to-end model based on long short-term memory (LSTM) encoder-decoder architecture. The model incorporates the LSTM encoder-decode architecture, the spatiotemporal convolutional neural network (STCNN), Word2Vec, and the Attention model. The STCNN captures continuously encoded motion information, Word2Vec converts words into word vectors for feature encoding, and the Attention model assigns weights to the target words. Based on the video dataset we built, we completed training and testing. Experiments have proved that the accuracy of the Mandarin Chinese lip-reading model is about 72%. Therefore, MCLRN can be used to identify the words spoken by the speaker.