Dynamic scheduling of mapreduce shuffle under bandwidth constraints

Abstract

Whether it is for e-science or business, the amount of data produced every year is growing at a high rate. Managing and processing those data raises new challenges. MapReduce is one answer to the need for scalable tools able to handle the amount of data. It imposes a general structure of computation and let the implementation perform its optimizations. During the computation, there is a phase called shuffle where every node sends a possibly large amount of data to every other node. This paper proposes and evaluates two algorithms to improve data transfers during the shuffle phase under bandwidth constraints.