An evolutionary methodology for handling data scarcity and noise in monitoring real events from social media data

Abstract

Every day text-based social media channels are flooded with millions of messages that comprise the most diverse topics. These channels are being used as a rich data source for monitoring different real world events such as natural disasters and disease outbreaks, to name a few. However, depending on the event being investigated, this monitoring may be severely affected by data scarcity and noise, allowing just coarse grain analysis in terms of time and space, which lack the specificity necessary for supporting actions at the local level. In this context, we present a methodology to handle data scarcity and noise while monitoring real world events using social media data in a fine grain. We apply our methodology to dengue-related data from Brazil, and show how it could improve significantly the performance of event monitoring at a local scale almost doubling the correlation observed in some cases.