Speech Training System for Hearing Impaired Individuals Based on Automatic Lip-Reading Recognition

Abstract

Using automatic lip recognition technology to promote social interaction and integration of hearing impaired individuals and dysphonic people is one of promising applications of artificial intelligence in healthcare and rehabilitation. Due to inaccurate mouth shapes and unclear expressions, hearing impaired individuals and dysphonic people cannot communicate as normal people do. In this paper, a speech training system for hearing impaired individuals and dysphonic people is constructed using state-of-the-art automatic lip-reading technology which combines convolutional neural network (CNN) and recurrent neural network (RNN). We train their speech skills by comparing different mouth shapes between the hearing impaired individuals and normal people. The speech training system can be divided into four parts. Firstly, we create a speech training database that stores mouth shapes of normal people and corresponding sign language vocabulary. Secondly, the system implements automatic lip-reading through a hybrid neural network of the MobileNet and the Long-Short-Term Memory Networks (LSTM). Thirdly, the system finds correct lip shape matched by sign language vocabulary from the speech training database and compares the result with lip shapes of hearing impaired individuals. Finally, the system draws comparison data and similarity rate based on the size of lips of hearing impaired individuals, the angle of opening lips, and the differences between different lip shapes. Giving a standard lip-reading sequence for the hearing impaired for their learning and training. As a result, hearing impaired individuals and dysphonic people can analyze and correct their vocal lip shapes based on the comparison results. They can perform training independently to improve their mouth shape. Besides, the system can help hearing impaired individuals learn how to pronounce correctly with the help of medical devices such as cochlear implants. Experiments show that the speech training system based on automatic lip-reading recognition can effectively correct lip shape of the hearing impaired individuals while they speak and improve their speech ability without help from others.