Application development for hand gestures recognition with using a depth camera

Abstract

The aim of the work is to develop an application for hand gestures identification based on a convolutional neural network using the TensorFlow & Keras deep learning frameworks. The gesture recognition system consists of a gesture presentation, a gesture capture device (sensor), the preprocessing and image segmentation algorithms, the features extraction algorithm, and gestures classification. As a sensor, Intel® Real Sense™ depth camera D435 with USB 3.0 support for connecting to a computer was used. For video processing and extraction both RGB images and depth information from the input data, functions from the Intel Real Sense library are applied. For pre-processing and image segmentation algorithms computer vision methods from the OpenCV library are implemented. The subsystem for the features extracting and gestures classification is based on the modified VGG-16, with weights previously trained on the ImageNet database. Performance of the gesture recognition system is evaluated using a custom dataset. Experimental results show that the proposed model, trained on a database of 2000 images, provides high recognition accuracy (99.4%).