3D integral imaging based augmented reality with deep learning implemented by faster R-CNN

Abstract

In this paper, we propose a Marker-less AR method for 3D integral imaging displayed above the smart phone via an integral lenslet array. The marker-less AR is implemented by using the Deep Learning method to recognize an object detected by the smart phone device with Android platform. To obtain a real-time recognition speed we use the Faster R-CNN algorithm for the object recognition task. This system will start from the Android device that captures the image of the object the user wants to detect and sends it to the server. After the server receives the image, it starts to process the object recognition and saves the result into a database. When the database gets updated, the server sends back a feedback to the Android device. In the android system, a video file related to the content of the object it recognized begins to be played on the Android device. The video file is pre-processed so that it will appear as a 3D content when it is seen through a 3D lenslet array case which is covered above the Android device display. The integral imaging algorithm makes the pre-processed 3D content to produce a pop-up 3D/hologram. Applying this proposed method could make an application become more compatible with non-high-specs device.