DX-FloodLine: End-To-End Deep Explainable Pipeline for Real Time Flood Scene Object Detection From Multimedia Images

Abstract

In recent years, an alarming number of natural disasters, specifically hurricanes and floods have affected the United States. Flooding is a global crisis, however we still do not have an automated approach for real-time flood risk detection and mitigation. Detecting inundation in urban areas and road segments is crucial for vehicle routing and traffic management. Unlike remote sensing images that have been used in past studies, it is better to mine social media data and geo-tagged images or videos for flood estimation in near real time. In this paper, multimedia content was collected from social media streaming services, primarily Twitter for analysis. We propose DX-FloodLine, an interpretable, intelligent, multi-stage, end-to-end, deep neural network-based pipeline to classify near real time emerging flood occurrence from images and detect submerged objects and pedestrians around flooded regions. We introduce a novel hybrid neural network for flood detection, a VGG16(Visual Geometry Group 16-layer model)-LSTM (Long short term memory) ensemble. Our novel ensemble architecture recognizes flood in images in the first stage of pipeline, to be later passed on to second stage for object detection. We applied interpretable models to flood classifiers to identify model shortcomings and then incrementally train for continuous improvement. DX-FloodLine was deployed and tested on unseen and near real time streamed flood images. Our VGG16-LSTM flood recognition model achieved around 90% validation accuracy on multiple benchmark studies and surpassed other competitors by a good margin.