Large-Scale System for Social Media Data Warehousing: The Case of Twitter-Related Drug Abuse Events Integration

Abstract

Social media data has become an integral part in the business data and should be integrated into the decisional process for better decision making based on information which reflects better the true situation of business in any field. However, social media data are unstructured and generated in very high frequency which exceeds the capacity of the data warehouse. In this work, the authors propose to extend the data warehousing process with a staging area which is a large-scale system implementing an information extraction process using Storm and Hadoop frameworks to better manage their volume and frequency. Concerning structured information extraction, mainly events, the authors combine a set of techniques from NLP, linguistic rules, and machine learning. Finally, they propose the adequate data warehouse conceptual model for events modeling and integration with enterprise data warehouse using an intermediate table called Bridge table. For application and experiments, they focus on drug abuse events extraction from Twitter data and their modeling into the event data warehouse.