Hand gestures recognition model for augmented reality robotic applications

Abstract

Augmented reality (AR) is a research promising field. Its main idea is to integrate and merge the virtual world with the real world. Augmented reality could improve or enhance our perception of the real world by integrating virtual objects. The existing hand gesture applications related to augmented reality can detect hand motion or track it in addition to the ability to construct a 3D model of tracked hand using markers and motion sensing devices like Kinect, LeapMotion, and AR/VR instruments. In this paper, a hand gesture recognition model based on deep convolutional neural network is proposed to be used in 3D virtual environments for robotic teleportation. This model is tested on HTC VIVE Pro AR/VR instruments using the VIVE eye and on a Kinect v2 to control an industrial manipulator in real time using only the hand movements in both online and offline control modes.