Scalable sentiment analytics

Abstract

Spark has become a widely popular analytics framework that provides an implementation of the equally popular Map Reduce programming model. Hadoop is an Apache foundation framework that can be used for processing large datasets on a cluster of computers using the Map Reduce programming model. Mahout is an Apache foundation project developed for building scalable machine learning libraries, which includes built-in machine learning classifiers. In this paper, we show how to build a simple text classifier on Spark, Apache Hadoop, and Apache Mahout for extracting out sentiments from a text collection containing millions of text documents. Using a collection of 7 million movie reviews taken from IMDB, a Bayesian classifier was learned to predict sentiments for test reviews. Separate classifiers were learned on both Spark and Hadoop, i.e. our contenders for scalable sentiment analytics. Our empirical results showed that the sentiment learning task on Spark ran almost 10 times faster than the learning task on Hadoop.