Designing Framework for Real Time Twitter Data Analytics using Apache Flume and Pig

Abstract

In the world of technology, people prefer social media to express themselves. Record says Twitter has more than 321 million active users with 100 million users posting approximately 340 million tweets a day. Twitter is the largest source of breaking news on social issues specially election-related where people can express their views also suggest their opinion. Twitter is generating unlimited unstructured text data. Hadoop is one of the finest tools accessible for analyzing twitter data because it supports processing of distributed big data, streaming data, time stamped data, text data etc. Whereas Apache Flume is used to extract real time twitter data into HDFS. This study attempts to establish an analytical framework to derive and interpret structured as well as unstructured Twitter data. The proposed framework comprises of real time twitter data insertion, its processing, and data visualization utilizing Apache Flume and pig. In this project we fetch positive and negative tweets on election data from twitter and analyzing the party status and the probability to win the election.