Image fusion and super-resolution with convolutional neural network

Abstract

Image fusion aims to integrate multiple images of the same scene into an artificial image which contains more useful information than any individual one. Due to the constraints of imaging sensors and signal transmission broadband, the resolution of most source images is limited. In traditional processing framework, super-resolution is conducted to improve the resolution of the source images before the fusion operations. However, those super-resolution methods do not make full use of the multi-resolution characteristics of images. In this paper, a novel jointed image fusion and super-resolution algorithm is proposed. Source images are decomposed into undecimated wavelet (UWT) coefficients, the resolution of which is enhanced with convolutional neural network. Then, the coefficients are further integrated with certain fusion rule. Finally, the fused image is constructed from the combined coefficients. The proposed method is tested on multi-focus images, medical images and visible light and near infrared ray images respectively. The experimental results demonstrate the superior performances of the proposed method.