Data-driven framework for high-accuracy color restoration of RGBN multispectral filter array sensors under extremely low-light conditions

Abstract

RGBN multispectral filter array provides a cost-effective and one-shot acquisition solution to capture well-aligned RGB and near-infrared (NIR) images which are useful for various optical applications. However, signal responses of the R, G, B channels are inevitably distorted by the undesirable spectral crosstalk of the NIR bands, thus the captured RGB images are adversely desaturated. In this paper, we present a data-driven framework for effective spectral crosstalk compensation of RGBN multispectral filter array sensors. We set up a multispectral image acquisition system to capture RGB and NIR image pairs under various illuminations which are subsequently utilized to train a multi-task convolutional neural network (CNN) architecture to perform simultaneous noise reduction and color restoration. Moreover, we present a technique for generating high-quality reference images and a task-specific joint loss function to facilitate the training of the proposed CNN model. Experimental results demonstrate the effectiveness of the proposed method, outperforming the state-of-the-art color restoration solutions and achieving more accurate color restoration results for desaturated and noisy RGB images captured under extremely low-light conditions.