Hand Gesture Recognition for User Interaction in Augmented Reality (AR) Experience

Abstract

Augmented Reality (AR) has gained a lot of attraction in the recent past. Arguably, the most important tool that can make AR a household gadget is its interaction with the user. This may lead to two possible interaction methodologies: (i) Using an extra device for interaction; (ii) Using human hands as interaction. Former is probably the easy method, but it may increase the cost of the AR device, limiting its target users. Therefore, hand gestures are a feasible and efficient mode of interaction. However, for accurate and pleasant interaction, the AR device should be capable of hand gesture understanding. In this direction, we propose a hand gesture classification method, based on Convolutional Neural Networks (CNNs) that takes advantage of the pre-trained network weights for faster and efficient training, which also helps improve the quality of gesture classification. Moreover, the proposed approach takes advantage of hand detection for background elimination and efficient gesture recognition. The proposed approach is evaluated on the Hand gesture classification task for three datasets that differ in terms of the number of data samples, amount of gestures, and data quality. The obtained results show that our method outperforms state-of-the-art methods in most of the experimentation cases.