An Exploratory Survey of Hadoop Log Analysis Tools

Abstract

This paper aims to explore more about Hadoop Log Analysis tools and how they can assist with technological breakthroughs. Since the usage of clusters in large-scale computation is increasing, maintaining the quality of these clusters is important. The significance of monitoring and controlling the network is illuminated by this. There have been several methods that can manage the Hadoop cluster effectively [1]. The bulk of these tools collect appropriate information in each cluster node and submit it to be processed. The majority of these diagnostic methods are post-processing research tools. An exploratory assessment of these tools is presented in this paper. A typical application for a first Hadoop program is log analysis. Indeed, one of Hadoop's first applications was for the large-scale study of clickstream logs-logs that document details regarding the web pages users visit and in what order they access them. The data logs created by the IT infrastructure are also called data exhaust. Like smoke emanating from a running engine's exhaust pipe, a log is a by-product of a running system. The term "data exhaust" connotes pollution or waste, as several businesses would certainly treat this type of data with that in mind [1]. The use of the internet is increasing every day in the modern world. Data are obtained from numerous sources including sensor data, web search page logs, social network pages, visual photographs and recordings, transaction details, and GPS signals on mobile phones. Managing this kind of data is a time-consuming process. This type of unstructured data is referred to as Big Data. Hadoop is the basis for conceptualizing Big Data and addressing the challenge of getting it useful for analytics.