Efficient gesture recognition in Indian sign language using SENet fusion of multimodal data

Abstract

The integration of surface electromyography (sEMG) signals and accelerometer data has opened new avenues for precise recognition of complex gestures, particularly in applications like Indian Sign Language (ISL). This research proposes a Parallel Squeeze-and-Excitation Network (Parallel SENet) architecture to classify ISL gestures using high-dimensional multimodal inputs of sEMG and tri-axis accelerometer features. Unlike conventional deep learning approaches, this study introduces a unique parallel processing pipeline that utilizes distinct Squeeze-and-Excitation (SE) blocks for each modality, enhancing feature relevance through adaptive recalibration. This ensures effective feature extraction and improves intermodal interaction for robust classification. This study aims to improve the classification accuracy of ISL gestures by effectively handling the challenges associated with high-dimensional data and multimodal feature fusion. The methodology involves preprocessing multimodal data, designing parallel SE blocks for each input type, and fusing the recalibrated features into a shared representation space for classification. A comparative analysis was performed using state-of-the-art architectures, such as Convolutional Neural Networks (CNNs) and Dense Neural Networks (DNNs), to demonstrate the benefits of the proposed approach. The experimental results indicate that the Parallel SENet attains an impressive classification accuracy of 94.17%, surpassing other models in both precision and recall. These findings highlight the effectiveness of the SE mechanism in refining multimodal features and its scalability for large, high-dimensional datasets.