Utility-driven share scheduling algorithm in Hadoop

Abstract

Job scheduling in hadoop is a hot topic, however, current research mainly focuses on the time optimization in scheduling. With the trend of providing hadoop as a service to the public or specified groups, more factors should be considered, such as time and cost. To solve this problem, we present a utility-driven share scheduling algorithm. Considering time and cost, algorithm offers a global optimization scheduling scheme according to the workload of the job. Furthermore, we present a model that can estimate job execute time by cost. Finally, we implement the algorithm and experiment it in a hadoop cluster. © 2013 Springer-Verlag Berlin Heidelberg.