Resource aware adaptive scheduler for heterogeneous workload with task based job sampling

Abstract

Resource aware adaptive scheduling for Mapreduce jobs aims at improving resource utilization across machines. Mapreduce schedulers mainly have fixed number of execution slot on each tasktracker that represents the capacity of cluster. Here a method of dynamically adjusting the number of slots on tasktracker based on task completion gaol is implemented to maximize the resource utilization. A method of task based job sampling is used to get job profile information that inturn used to adjust the slots dynamically. Accuracy of our estimations where assessed based on completion time goal and actual execution time.