A big data reference architecture for teaching social media mining

Abstract

The analysis of big data represents an important capability for companies and in research and teaching. Data scientists, confronted with complex system configuration and implementation tasks, require affordable and state-of-the-art solutions, which are flexibly configurable to enable diverse analytical research scenarios. In this research, we describe an architecture for the collection, preprocessing, and analysis of social media data based on Hadoop, which we used in a master-level course. We demonstrate how to configure and integrate different components of the Hadoop/Spark ecosystem in order to manage the collection of large data volumes as social media data streams over Web APIs, distributed data storage, the definition of schemas, data preprocessing, and feature extraction, as well as the calculation of descriptive statistics and predictive models. Three exemplary student projects, shortly described in this paper, demonstrate the versatility of the presented solution. Our results can serve as a blueprint for similar endeavors at other educational institutions.