Hybrid fusion-based background segmentation in multispectral polarimetric imagery

Abstract

Multispectral Polarimetric Imagery (MSPI) contains significant information about an object's distribution, shape, shading, texture and roughness features which can distinguish between foreground and background in a complex scene. Due to spectral signatures being limited to material properties, Background Segmentation (BS) is a difficult task when there are shadows, illumination and clutter in a scene. In this work, we propose a two-fold BS approach: multiband image fusion and polarimetric BS. Firstly, considering that the background in a scene is polarized by nature, the spectral reflectance and correlations and the textural features of MSPI are calculated and analyzed to demonstrate the fusion significance. After that, integrating Principal Component Analysis (PCA) with Fast Fourier Transform (FFT), a hybrid fusion technique is proposed to show the multiband fusion effectiveness. Secondly, utilizing the Stokes vector, polarimetric components are calculated to separate a complex scene's background from its foreground by constructing four significant foreground masks. An intensity-invariant mask is built by differentiating between the median filtering versions of unpolarized and polarized images. A strongly unpolarized foreground mask is also constructed in two different ways, through analyzing the Angle of Linear Polarization (AoLP) and Degree of Linear Polarization (DoLP). Moreover, a strongly polarized mask and a strong light intensity mask are also calculated based on the azimuth angle and the total light intensity. Finally, all these masks are combined, and a morphological operation is applied to segment the final background area of a scene. The proposed two-fold BS algorithm is evaluated using distinct statistical measurements and compared with well-known fusion methods and BS methods highlighted in this paper. The experimental results demonstrate that the proposed hybrid fusion method significantly improves multiband fusion quality. Furthermore, the proposed polarimetric BS approach also improves the mean accuracy, geometric mean and F1-score to 0.95, 0.93 and 0.97, respectively, for scenes in the MSPI dataset compared with those obtained from the methods in the literature considered in this paper. Future work will investigate mixed polarized and unpolarized BS in the MSPI dataset with specular reflection.