Leveraging multimodal deep learning for natural disaster event classification and its damage severity analysis through social media posts

Abstract

Accurate and timely information is critical to effectively coordinate disaster response. Due to the diversity and complexity of data sources, it is difficult for traditional methods to classify disaster events and assess damage severity. Previous studies have mainly focused on specific tasks, such as information collection or humanitarian assistance, but have not adequately addressed the assessment of disaster loss severity. This paper proposes a hybrid learning model to improve disaster event classification and damage severity identification. The model combines image and text data, using ResNet50 to extract features from images, and using long short-term memory with an attention mechanism to learn sequences from text. This combination allows for a more contextual and informative representation of the input data. The experimental results shows that the proposed multimodal approach achieves significantly better results in disaster event classification attaining an accuracy of 90.31%, compared to existing methods. Furthermore, the model demonstrates promising capabilities in assessing damage severity, offering significant improvements for disaster management and response preparation where accuracy and dependability are crucial.