Applying LSTM and GRU Methods to Recognize and Interpret Hand Gestures, Poses, and Face-Based Sign Language in Real Time

Abstract

This research uses a real-time, human-computer interaction application to examine sign language recognition. This work develops a rule-based hand gesture approach for Indonesian sign language in order to interpret some words using a combination of hand movements, mimics, and poses. The main objective in this study is the recognition of sign language that is based on hand movements made in front of the body with one or two hands, movements which may involve switching between the left and right hand or may be combined with mimics and poses. To overcome this problem, a research framework is developed by coordinating hand gestures with poses and mimics to create features by using holistic MediaPipe. To train and test data in real time, the long short time memory (LSTM) and gated recurrent unit (GRU) approaches are used. The research findings presented in this paper show that hand gestures in real-time interactions are reliably recognized, and some words are interpreted with the high accuracy rates of 94% and 96% for the LSTM and GRU methods, respectively.