Effective utilization of storage space by applying file level and block-level deduplication over HDFS

Abstract

Hadoop framework is very efficient and easy to handle huge records storage as well as its processing. Hadoop makes use of massive commodity hardware clusters to save and process massive data in an allotted fashion. Open Source, Massive information handling capabilities and faster processing abilities made it very popular. Existing Hadoop Framework destroys metadata of preceding jobs, it actually allocates Data Nodes via ignoring what it has processed earlier and hence for each new process it reads data from all Data Nodes. There isn't any provision made for checking relationships between similar data blocks. Thus it weakens the Hadoop overall performance. The uploaded big data files are partitioned in to number of blocks and are distributed over node clusters. To avoid random block distribution and data-duplication, deduplication system is used. Such deduplication system focuses on space management and only keeps track of data files on Hadoop Distributed File System (HDFS). Such system do not participate in efficient job execution in map reduce environment. For efficient execution of job, data locality information and job metadata is stored. Time required for job execution can be decreased for next execution of same job by preserving job metadata. A combined environment produce efficient job execution results with efficient space management.