Visual–inertial fusion-based registration between real and synthetic images in airborne combined vision system

Abstract

Combined vision system is a perspective display concept to enhance a situation awareness of the pilots during aircraft landing, which integrates a real 2-D image captured from forward-looking infrared camera with a synthetic 3-D image derived by the aircraft pose and terrain database. However, the inertial measured errors significantly affect the conformal display of combined vision. This article proposes a novel method for real and synthetic images registration based on visual–inertial fusion. It includes the following key steps: (1) detect and extract the real runway features from forward-looking infrared image; (2) generate the synthetic runway features simultaneously; (3) set up vision measurement model with real and synthetic runway features; (4) integrate inertial data and visual observations in the square-root unscented Kalman filter; (5) create a synthetic 3-D scene by the filtered pose data and integrate it with a real 2-D image. The experimental results demonstrate that our method can guarantee the conformal display of combined vision system in GPS-denied and low visibility conditions.