Novel Approach for Augmented Reality using Convolutional Neural Networks

Abstract

In this paper, we will exploit the potential of convolutional neural networks (CNN) in augmented reality. Our work combines existing approaches and produces a new method for aligning a virtual object in the real world and in real time. Our method consists in detecting 2D objects of one or more classes present in the real world with CNN algorithms, then using the output of the network to calculate the position of the camera with the PnP algorithm in order to augment the detected object with additional information. The lightness of the MobileNet convolutional neural network combined with the speed of the Single Shot multibox Detector (SSD) framework allows to analyze the acquired images in real time and to use devices with limited resources and performance. We use a trained model that detects 20 different classes, the network receives as input an image sequence acquired in real time. The output of the network provides the set of detected classes as well as the coordinates of the corners of the surrounded rectangle on the object of this class. The coplanar coordinates of this rectangle are used to calculate the camera position and to align a 3D virtual object in the middle of the bounding box surrounding the detected object. The results obtained in the experimental part show the importance and the robustness of the method.