A modified key partitioning for BigData using MapReduce in Hadoop

Abstract

In the period of BigData, massive amounts of structured and unstructured data are being created every day by a multitude of everpresent sources. BigData is complicated to work with and needs extremely parallel software executing on a huge number of computers. MapReduce is a current programming model that makes simpler writing distributed applications which manipulate BigData. In order to make MapReduce to work, it has to divide the workload between the computers in the network. As a result, the performance of MapReduce vigorously depends on how consistently it distributes this study load. This can be a challenge, particularly in the arrival of data skew. In MapReduce, workload allocation depends on the algorithm that partitions the data. How consistently the partitioner distributes the data depends on how huge and delegate the sample is and on how healthy the samples are examined by the partitioning method. This study recommends an enhanced partitioning algorithm using modified key partitioning that advances load balancing and memory utilization. This is completed via an enhanced sampling algorithm and partitioner. To estimate the proposed algorithm, its performance was compared against a high-tech partitioning mechanism employed by TeraSort. Experimentations demonstrate that the proposed algorithm is quicker, more memory efficient and more accurate than the existing implementation.