Robustness analysis of 3D convolutional neural network for human hand gesture recognition

Abstract

Recently, a number of methods for dynamic hand gesture recognition has been proposed. However, deployment of such methods in a practical application still has to face with many challenges due to the variation of view point, complex background or subject style. In this work, we deeply investigate performance of advanced convolutional neural networks for a specific case of hand gestures and evaluate how robust it is to above variations. To this end, we adopt an existing 3D convolutional neural network which was originally proposed for general human action recognition and obtained very competitive accuracy. We extend it to two-streams architecture (RGB and optical flow) and apply transfer learning on our dataset of hand gestures. To evaluate the robustness of the method, we design carefully a multi-view dataset that composes of five dynamic hand gestures in indoor environment with complex background. Experiments with single or cross view on this dataset show that background and viewpoint has strong impact on recognition robustness. In addition, the network's performances are mostly increased by multi-modality combinations and fine-tuning strategy. This analysis helps to make recommendation for deploying the method in real situation.