Application of neural networks ensemble method for the Kazakh sign language recognition

Abstract

Sign languages are an extremely important means of communication in many cases, especially for deaf and hard of hearing people. But the same gesture can convey different meanings in different countries, so many different sign languages have been developed all over the world. In this study, a convolutional neural network (CNN) model was developed based on an ensemble method containing the ResNet-50 and VGG-19 architectures, which will be able to classify the Kazakh sign language (KSL) consisting of 42 Kazakh alphabet signs (classes). A proprietary data set of 57,708 images for 42 signs of the KSL has been formed. The ensemble model was compared with ResNet-50 and VGG-19 by evaluation metrics such as accuracy, precision, recall, f1-measure, and loss function. The recognition accuracy of the ensemble method reached 95.7%, exceeding the performance of ResNet-50 and VGG-19. The developed method was also tested on test data, where 35 out of 42 gestures were recognized completely correctly. The reliability of the proposed approach and the classification results obtained by using preprocessing methods and data augmentation techniques to expand the data set was confirmed by a computational experiment.