The GAN Spatiotemporal Fusion Model Based on Multiscale Convolution and Attention Mechanism for Remote Sensing Images

Abstract

High spatial and temporal resolution remote sensing images are essential for monitoring vegetation, natural disasters, and changes in the ground surface. However, acquiring such images is challenging due to current technical limitations and cost constraints. Spatiotemporal fusion offers an effective and economical solution to achieve high spatial and temporal resolution simultaneously. This article introduces a new generative adversarial network (GAN) spatiotemporal fusion model based on multiscale convolution and attention mechanism for remote sensing images (MSCAM-GAN), to generate high-resolution fused images. The generator in MSCAM-GAN comprises three key components: feature extraction, feature fusion, and image reconstruction. Employing an encoder-decoder architecture, the generator effectively extracts multilevel features, accommodating significant resolution differences between high-resolution and low-resolution images. In the feature extraction stage, multiscale convolutional attention network (MSCAN) captures detailed features across multiple scales, dealing with spatial dependencies and long-distance relationships within the images. During the feature fusion stage, a dual parallel attention feature fusion mechanism is designed to fully integrate the extracted multiscale features. Different attention weights are assigned based on their contributions to the final output, resulting in more accurate predicted images. MSCAM-GAN was tested on the Coleambally irrigated area and lower Gwydir catchment datasets and compared with classic spatiotemporal fusion algorithms. Ablation experiments were conducted to evaluate the effectiveness of the various submodules in MSCAM-GAN. Experimental results and ablation analysis demonstrate the superior performance of the proposed method compared to other approaches.