Deep fusion of hyperspectral images and multi-source remote sensing data for classification with convolutional neural network

Abstract

Hyperspectral images (HSIs) have abundant spectral characteristics. However, their spatial resolution is relatively low. Some remote sensing data have complementary advantages with HSIs, such as the LiDAR, which can provide elevation information, and the high spatial resolution data, which have precise spatial information. The combination of HSIs with the multi-source remote sensing data for fusion classification can make up the deficiency of tits relatively low spatial resolution. In recent years, deep learning-based methods have been investigated for hyperspectral remote sensing classification and have made breakthroughs. Meanwhile, the feature extraction process of the deep network is an independent process. Therefore, it may not obtain the most beneficial features for classification accurately and may influence the classification accuracy. At the same time, the techniques may not perform well when using limited training samples in HSIs because of massive parameters and complex network structure.Aiming at this problem, the frequently used traditional features of remote sensing data for classification are discussed in this paper. A new deep learning-based feature level data fusion classification framework that integrates traditional textural features into Convolutional Neural Network (CNN) approach (T-F-CNN) is proposed for the accurate fusion classification of HSIs and multi-source remote sensing data. The proposed method can be implemented in three steps. First, the traditional features are extracted from the HSIs or the multi-source remote sensing data. Second, CNN are built. The original HSIs, the original multi-source remote sensing data, and the traditional features, which are obtained in the first step, are inputted into the CNN of the deep feature extraction. Finally, the deep features obtained in the second step are concatenated in a concatenate layer of CNN, and SoftMax is used to generate classification maps at the end of the framework.Result The proposed classification scheme is tested on two data sets, namely, Houston and Thetford Mines Area data sets. The proposed T-F-CNN is compared with the pixel-level methods, such as Support Vector Machines (SVM) with the Radial Basis Function (RBF) (T-P-SVM), the CNN fusion method (P-CNN), and the CNN with traditional features (T-P-CNN); and the feature-level methods, such as CNN fusion method (F-CNN) and CNN method combined with original traditional features (T'-F-CNN). On both data sets, the proposed method shows a higher classification accuracy than other methods. Meanwhile, when the training samples reach a minimum number, the proposed method could provide the highest overall classification accuracy.The results obtained by the proposed method on the two real hyperspectral data sets demonstrate that the classification accuracy can be improved. Furthermore, the proposed T-F-CNN method outperforms some traditional deep learning methods and exhibits higher computing efficiency than a few advanced deep learning techniques.