Using LSTM to translate Thai sign language to text in real time

Abstract

Between 2019 and 2022, as the Covid-19 pandemic unfolded, numerous countries implemented lockdown policies, leading most corporate companies to permit employees to work from home. Communication and meetings transitioned to online platforms, replacing face-to-face interactions. This shift posed challenges for deaf or hearing-impaired individuals who rely on sign language, using hand gestures for communication. However, it also affected those who can hear clearly but lack knowledge of sign language. Unfortunately, many online meeting platforms lack sign language translation features. This study addresses this issue, focusing on Thai sign language. The objective is to develop a model capable of translating Thai sign language in real-time. The Long Short-Term Memory (LSTM) architecture is employed in conjunction with MediaPipe Holistic for data collection. MediaPipe Holistic captures keypoints of hand, pose, and head, while the LSTM model translates hand gestures into a sequence of words. The model’s efficiency is assessed based on accuracy, with real-time testing achieving an 86% accuracy, slightly lower than the performance on the test dataset. Nonetheless, there is room for improvement, such as expanding the dataset by collecting data from diverse individuals, employing data augmentation techniques, and incorporating an attention mechanism to enhance model accuracy.