Multiscale Spatial Fusion and Regularization Induced Unsupervised Auxiliary Task CNN Model for Deep Super-Resolution of Hyperspectral Images

Abstract

Hyperspectral images (HSI) feature rich spectral information in many narrow bands but at a cost of a relatively low spatial resolution. As such, various methods have been developed for enhancing the spatial resolution of the low-resolution HSI (Lr-HSI) by fusing it with high-resolution multispectral images (Hr-MSI). The difference in spectrum range and spatial dimensions between the Lr-HSI and Hr-MSI has been fundamental but challenging for multispectral/hyperspectral (MS/HS) fusion. In this article, a multiscale spatial fusion and regularization induced auxiliary task based convolutional neural network model is proposed for deep super-resolution of HSI, where an Lr-HSI is fused with an Hr-MSI to reconstruct a high-resolution HSI (Hr-HSI) counterpart. The multiscale fusion is used to efficiently address the discrepancy in spatial resolutions between the two inputs. Based on the general assumption that the acquired Hr-MSI and the reconstructed Hr-HSI share similar underlying characteristics, the auxiliary task is proposed to learn a representation for improved generality of the model and reduced overfitting. Experimental results on five public datasets have validated the effectiveness of our approach in comparison with several state-of-the-art methods.