Musical Instrument Performance in Augmented Virtuality

Abstract

In this paper, we present the preliminary study results about musical instrument performance in Augmented Virtuality (AV). In our proposed approach, musician can prerecord their performance using any camera device. Then, through the processing pipeline that we have proposed, the musicians are able to play musical instruments in entirely virtual worlds. According to the 3D virtual scene set up for Virtual Reality (VR) glasses, the audience can watch the performance in virtual reality where the musician is playing music in the real world. In the end, we rendered the musician's performance into 360 VR video to enhance the immersiveness of the user experiences.