Deep-Learning-Based Object Detection and Tracking of Debris Flows in 3-D Through LiDAR-Camera Fusion

Abstract

Debris flows are mixtures of water and sediments and are a significant natural hazard in mountainous regions. A novel monitoring approach combining LiDAR sensors and video cameras enables detailed observations of debris-flow dynamics, achieving high spatial (<2 cm) and temporal (10 Hz) resolution in the field. This technology allows for analyzing individual objects within the flow in unprecedented detail, essential for understanding debris-flow behavior. However, generating large datasets and enabling real-time monitoring requires automated detection and tracking of fast-moving debris-flow features. In this study, deep-learning algorithms for object detection (YOLOv5 and YOLOv8) were applied to optical images to identify orders of magnitude more debris-flow features than with existing approaches, such as surge waves (SWs) and boulders. Projecting these detections onto the LiDAR point clouds enabled the extraction of kinematic and geometric properties, including velocity and object size. The surge-wave detector performed exceptionally well, with mean average precisions (mAP) above 0.9. While smaller features like boulders and wood were more difficult to detect, the small feature (SF) detector’s mAP still exceeded 0.7. Testing the detectors at different channel locations showed that a multidomain detector can reduce labeling efforts by over 75% while maintaining performance. We further demonstrated that SWs accelerate smaller features like wood, and used the dataset to calculate a wave-breaking index (adapted from coastal engineering) with a threshold of 0.4. The developed methods represent a step forward in automated debris-flow monitoring, offering scalable tools for real-time hazard assessment and deeper insight into debris-flow dynamics.