RA-Net: reverse attention for generalizing residual learning

Abstract

Since residual learning was proposed, identity mapping has been widely utilized in various neural networks. The method enables information transfer without any attenuation, which plays a significant role in training deeper networks. However, interference with unhindered transmission also affects the network’s performance. Accordingly, we propose a generalized residual learning architecture called reverse attention (RA), which applies high-level semantic features to supervise low-level information in the identity mapping branch. It means that higher semantic features selectively transmit low-level information to deeper layers. In addition, we propose a Modified Global Response Normalization(M-GRN) to implement reverse attention. RA-Net is derived by embedding M-GRN in the residual learning framework. The experiments show that the RA-Net brings significant improvements over residual networks on typical computer vision tasks. For classification on ImageNet-1K, compared with resnet101, RA-Net improves the Top-1 accuracy by 1.7% with comparable parameters and computational cost. For COCO detection, on Faster R-CNN, reverse attention improves box AP by 1.9%. Meanwhile, reverse attention improves UpperNet’s mIoU by 0.7% on ADE20K segmentation.