Integrating the First Person View and the Third Person View Using a Connected VR-MR System for Pilot Training

Abstract

Virtual reality (VR)-based flight simulator provides pilots the enhanced reality from the first-person view. Mixed reality (MR) technology generates effective 3D graphics. The users who wear the MR headset can walk around the 3D graphics to see all its 360 degrees of vertical and horizontal aspects maintaining the consciousness of real space. A VR flight simulator and an MR application were connected to create the capability of both first-person view and third-person view for a comprehensive pilot training system. This system provided users the capability to monitor the aircraft progress along the planned path from the third-person view as well as how the aircraft pilot follows the desired flight procedures in the cockpit from the first-person view. Six flight instructors provided their feedback after trying the prototype functions. The three flight instructors with more than 1,000 flight hours gave negative feedback whereas the three instructors with less than 1,000 flight hours were more open to the technologies. The effectiveness of MR-based real-time monitoring and post-debriefing system is yet to be clear. However, the test results included the potentials of the prototype for future improvement based on MR technology’s flexibility.