SCSA: Exploring the synergistic effects between spatial and channel attention

Abstract

Channel and spatial attentions have respectively brought significant improvements in extracting feature dependencies and spatial structure relations for various downstream vision tasks. The combined use of both channel and spatial attentions is widely considered beneficial for further performance improvement; however, the synergistic effects between channel and spatial attentions, especially in terms of spatial guidance and mitigating semantic disparities, have not yet been thoroughly studied. This motivates us to propose a novel Spatial and Channel Synergistic Attention module (SCSA), entailing our investigation toward the synergistic relationship between spatial and channel attentions at multiple semantic levels. Our SCSA consists of two parts: the Shareable Multi-Semantic Spatial Attention (SMSA) and the Progressive Channel-wise Self-Attention (PCSA). SMSA integrates multi-semantic information and utilizes a progressive compression strategy to inject discriminative spatial priors into PCSA's channel self-attention, effectively guiding channel recalibration. Additionally, the robust feature interactions based on the Channel-wise single-head self-attention mechanism in PCSA further mitigate the disparities in multi-semantic information among different sub-features within SMSA. We conduct extensive experiments on seven benchmark datasets, including classification on ImageNet-1K, object detection on MSCOCO, segmentation on ADE20K, and four other complex scene detection datasets. Our results demonstrate that our proposed SCSA not only surpasses the current plug-and-play state-of-the-art attention but also exhibits enhanced generalization capabilities across various task scenarios. The code and models are available at: https://github.com/HZAI-ZJNU/SCSA.