MROrder: Flexible job ordering optimization for online MapReduce workloads

Abstract

MapReduce has become a widely used computing model for large-scale data processing in clusters and data centers. A MapReduce workload generally contains multiple jobs. Due to the general execution constraints that map tasks are executed before reduce tasks, different job execution orders in a MapReduce workload can have significantly different performance and system utilization. This paper proposes a prototype system called MROrder to dynamically optimize the job order for online MapReduce workloads. Moreover, MROrder is designed to be flexible for different optimization metrics, e.g., makespan and total completion time. The experimental results show that MROrder is able to improve the system performance by up to 31% for makespan and 176% for total completion time. © 2013 Springer-Verlag.