Optimizing Search and Data Analytics of Twitter Data using Elastic Search Algorithms

Abstract

Quick data acquisition and analysis became an important tool in the contemporary era. Real time data is made available in World Wide Web (WWW) and social media. Especially social media data is rich in opinions of people of all walks of life. Searching and analysing such data provides required business intelligence (BI) for applications of various domains in the real world. The application may be in the area of politics or banking or insurance or healthcare industry. With the emergence of cloud computing, volumes of data are added to cloud storage infrastructure and it is growing exponentially. In this context, Elasticsearch is the distributed search and analytics engine that is very crucial part of Elastic Stack. For data collection, aggregation and enriching it Beats and Logstash are used and such data is stored in Elasticsearch. For interactive exploration and visualization Kibana is used. Elasticsearch helps in indexing of data, searching efficiently and performing data analytics. In this paper, the utility of Elasticsearch is evaluated for optimising search and data analytics of Twitter data. Empirical study is made with the Elasticsearch tool configured for Windows and also using Amazon Elasticsearch and the results are compared with state of art. The experimental results revealed that the Elasticsearch performs better than the existing ones.