Scrutinize the Idea of Hadoop-Based Data Lake for Big Data Storage

Abstract

Data is the driving force for the economy of any country. Data works as a fuel to the economy. The prime’s task for the organization is to store the data and use that data for the decision-making process. In the past, the organization used data warehouse and data marts to store the data and use for decision-making purpose, but for technology advancement, data warehouse faces many challenges and it fails to fulfill market demands. The biggest challenge for the data warehouse is to manage big data, data with velocity, data with huge volume, data with variety, data with veracity, and data with value. As of twenty-first century starts, worlds witness many new technologies like AI, deep learning, machine learning, and many more which all completely depend upon big data. Data warehouse fails to fulfill data engineer requirements to use these technologies to make decision-making system more effective. Data engineers want a new repository to store big data as data warehouse works on the concept of schema-on-write state that transforms the data before storage but engineers want data in raw format and later on according to business need they can transform the data to get the different values from data. To overcome the challenges which were faced by the data warehouse, research comes with a new concept known as data lake, a technologically advanced version of a data warehouse. Data lake works on the concept of schema-on-read. The objective of this chapter is to examine the idea of data lake from a user perspective as well as a technology perspective.