S3Net: Spectral-Spatial-Semantic Network for Hyperspectral Image Classification with the Multiway Attention Mechanism

Abstract

In hyperspectral image (HSI) classification, it is a great challenge on how to extract key informative spectral-spatial features efficiently and suppress useless features from abundant spectral-spatial information. In this article, inspired by the attention mechanism of the human visual system, we propose a novel spectral-spatial-semantic network (S3Net) with the multiway attention mechanism for HSI classification. The S3Net consists of a spectral branch, a spatial branch, and a multiscale semantic module. The spectral branch extracts the multiway spectral features with a dense spectral block and a multiway spectral attention module. The spatial branch extracts the multiway spatial features with a multiscale spatial block and a multiway spatial attention module. The multiscale semantic module extracts spectral-spatial-semantic features that are used for classification. In the proposed S3Net, the multiway attention modules in spectral and spatial branches are built to enhance the extraction ability of the key informative spectral-spatial features. The multiscale spatial block in the spatial branch is designed to learn strong complementary and related information. The Res2Net in the multiscale semantic module is used to learn multiscale semantic features at a granular level. A large number of experimental results demonstrate that, on the University of Pavia, Kennedy Space Center, and Pavia Center data sets, the proposed S3Net achieves higher classification accuracy than state-of-the-art methods on the limited training samples. Remarkably, our S3Net also achieves the best performance on the mineral exploration HSI data set, called Huoshaoyun, which is collected by the GaoFen-5 (GF-5) satellite.