AI-powered real-time hand sign recognition: enhancing communication for the hearing- and speech-impaired community

Abstract

This study presents a real-time hand-sign recognition system to support communication for persons with hearing and speech impairments, with a focus on one-handed finger-spelling (the “dactyl alphabet”). The method performs on-device, frame-based detection and classification using RetinaNet with a ResNet-50 backbone, chosen for its favourable speed–accuracy trade-off and stable training on modest, class-imbalanced datasets typical of sign recognition. To ensure robustness beyond laboratory conditions, we curated a purpose-built dataset of more than 5600 annotated images captured across varied indoor and outdoor scenes, lighting conditions, backgrounds, camera distances, skin tones, and hand shapes. In a user study with 30 participants, the system achieved an average accuracy exceeding 93% across environments while maintaining interactive latency suitable for everyday use. In contrast to prior work that relies on controlled public corpora and offline pipelines, this work couples an in-the-wild, application-specific dataset with a real-time, on-device pipeline validated in realistic settings. Current limitations include recognition of isolated, single-hand signs rather than continuous sequences; future work will address sequence modelling and user-specific personalization. These results indicate that accurate, low-latency sign recognition is feasible on mainstream hardware and practically deployable beyond the lab.