An Automated Framework for Abnormal Target Segmentation in Levee Scenarios Using Fusion of UAV-Based Infrared and Visible Imagery

Abstract

Highlights: What are the main findings? We reframe levee monitoring as an unsupervised anomaly detection task, unifying diverse hazards and response elements into a single “abnormal targets” category for comprehensive situational awareness. We propose a novel, fully automated and training-free framework that integrates multi-modal fusion with an adaptive segmentation module, where Bayesian optimization automatically tunes a mean-shift algorithm. What is the implication of the main finding? Our framework provides a practical solution for a challenging, data-scarce domain by eliminating the need for labelled training data, a major bottleneck for traditional supervised methods. The proposed method demonstrates superior performance over all baselines in complex, real-world scenarios, proving the effectiveness of synergistic multi-modal fusion and adaptive unsupervised learning for disaster management. Levees are critical for flood defence, but their integrity is threatened by hazards such as piping and seepage, especially during high-water-level periods. Traditional manual inspections for these hazards and associated emergency response elements, such as personnel and assets, are inefficient and often impractical. While UAV-based remote sensing offers a promising alternative, the effective fusion of multi-modal data and the scarcity of labelled data for supervised model training remain significant challenges. To overcome these limitations, this paper reframes levee monitoring as an unsupervised anomaly detection task. We propose a novel, fully automated framework that unifies geophysical hazards and emergency response elements into a single analytical category of “abnormal targets” for comprehensive situational awareness. The framework consists of three key modules: (1) a state-of-the-art registration algorithm to precisely align infrared and visible images; (2) a generative adversarial network to fuse the thermal information from IR images with the textural details from visible images; and (3) an adaptive, unsupervised segmentation module where a mean-shift clustering algorithm, with its hyperparameters automatically tuned by Bayesian optimization, delineates the targets. We validated our framework on a real-world dataset collected from a levee on the Pajiang River, China. The proposed method demonstrates superior performance over all baselines, achieving an Intersection over Union of 0.348 and a macro F1-Score of 0.479. This work provides a practical, training-free solution for comprehensive levee monitoring and demonstrates the synergistic potential of multi-modal fusion and automated machine learning for disaster management.