On demand resource scheduler based on estimating progress of jobs in Hadoop

Abstract

In order to meet the need of setting deadline for Hadoop MapReduce job and improve resource utilization of Hadoop cluster, a resource scheduler based on collecting the running information of tasks is proposed. According to the information of resource usage, the progress of job, the deadline of job, and the handling time of job, we estimate the resource demand of jobs, and then schedule these jobs according to their resource demand. Meanwhile, a method to judge whether the resource of cluster can meet the deadline of all the jobs in cluster is proposed. When the jobs will miss the deadline under the allocated resources, scheduler applies to cloud platform for extra resources. Experimental results show the on demand resource scheduler can increase the utilization of resource in Hadoop cluster and approximately ensure the deadline of jobs.