MSMapper: An adaptive split assignment scheme for MapReduce

Abstract

MapReduce as a popular platform has been extensively used for solving data-intensive applications. A number of tuning parameters can be applied to improve the performance of MapReduce. Among these parameters, the number of map tasks (mappers) driven by the number of logical input splits has a dramatic effect on the performance. However, subject to one-to-one correspondence between mappers and splits, the tradeoff between mapper-level parallelism and mapper startup costs must be carefully evaluated based on the input size and the split size. Meanwhile, the manual parameter configuration is lack of flexibility to meet the performance requirements of different jobs. In this paper, an adaptive split assignment scheme is proposed to decouple the number of mappers from the number of splits. We introduce the MSMapper(Multi-Split Mapper), a modified self-tuning mapper in which multiple splits can be assigned to one mapper. And with aid of inter-MSMapper communication, we reveal the potential that map tasks can be constructed without dependence on the number of splits, while the modified MapReduce architecture can sustain fine-grained load balancing and fault tolerance, as well as coarse-grained task startup overhead. We built our prototype on top of the Hadoop MapReduce realization, and present a comprehensive evaluation that shows the benefits of the MSMapper in common scenarios where split sizing problems arise. The results show that the modified version can improve the performance by a factor of 2.5. © 2012 Springer-Verlag.