A multimodal deep fusion framework for highway traffic anomaly detection

Abstract

This paper presents a novel framework for detecting and predicting abnormal traffic events on highways. Traditional traffic monitoring systems often rely on a single data source, which limits detection accuracy and robustness in complex environments. To address these challenges, we propose a multimodal deep fusion framework based on heterogeneous graph neural networks (HGNNs), enhanced by an ensemble contrastive pessimistic likelihood estimation (CPLE) algorithm. The framework integrates both static and dynamic traffic data, including video images, traffic flow, vehicle speed, and tunnel weather conditions. Through effective feature fusion, it significantly improves the accuracy and real-time performance of anomaly detection. Experimental results show that the model performs robustly across various scenarios, accurately identifying abnormal traffic events with high precision and stability. Compared with existing models such as AGC-LSTM and AttentionDeepST, the proposed MHGNN-CPLE model demonstrates superior performance, particularly in static detection tasks, achieving an accuracy of 0.980 and an F1 score of 0.967. In contrast, AGC-LSTM and AttentionDeepST achieve 0.965/0.945 and 0.960/0.935 in accuracy and F1 score, respectively. In dynamic scenarios, the model also maintains high accuracy under varying noise levels, indicating strong robustness. The research is motivated by the growing challenges of urbanization, where real-time detection and prediction of traffic anomalies are increasingly critical. Our framework effectively integrates multimodal data and leverages HGNNs to capture complex spatiotemporal dependencies, while the CPLE algorithm enhances robustness under uncertainty. The results confirm that the proposed method offers a reliable and accurate solution for real-time traffic anomaly detection, representing a significant advancement in intelligent transportation systems.