Pengenalan Kata Kolok Secara Real Time Menggunakan Mediapipe dan Algoritma Long Short-Term Memory (LSTM)

Abstract

Sign language is the primary form of communication for individuals who are deaf or mute. In the Bengkala community, there exists a unique sign language known as Kolok. Modern technology enables this local knowledge to be integrated into recognition systems. Building on this opportunity, the present study aims to develop a recognition system for Kolok Sign Language using the Long Short-Term Memory (LSTM) algorithm. The dataset was obtained directly from the Bengkala Village community, collected through two methods: manual recording using a smartphone camera and real-time frame capture with OpenCV assisted by a webcam. Ten vocabulary categories were utilized in a dataset comprising 300 time-series videos, each consisting of 30 frames. MediaPipe was employed to analyze each frame and extract keypoint coordinates of body and hand positions. The features used for modeling were the x, y, and z coordinates of body poses and hand landmarks. The extracted data were then processed using a two-layer LSTM architecture. The training results demonstrated highly consistent performance. At epochs 500 and 550, the model achieved accuracies of 0.91 and 0.93, along with F1-scores of 0.92 and 0.93, respectively. The highest performance was recorded at epoch 450, with a precision of 0.95, recall of 0.96, and an F1-score of 0.94.