Capturing Node Resource Status and Classifying Workload for Map Reduce Resource Aware Scheduler

3Citations
Citations of this article
2Readers
Mendeley users who have this article in their library.
Get full text

Abstract

There has been an enormous growth in the amount of digital data, and numerous software frameworks have been made to process the same. Hadoop MapReduce is one such popular software framework which processes large data on commodity hardware. Job scheduler is a key component of Hadoop for assigning tasks to node. Existing MapReduce scheduler assigns tasks to node without considering node heterogeneity, workload type, and the amount of available resources. This leads to overburdening of node by one type of job and reduces the overall throughput. In this paper, we propose a new scheduler which capture the node resource status after every heartbeat, classifies jobs into two types, CPU bound and IO bound, and assigns task to the node which is having less CPU/IO utilization. The experimental result shows an improvement of 15-20 % on heterogeneous and around 10 % of homogeneous cluster with respect to Hadoop native scheduler. © Springer India 2015.

Cite

CITATION STYLE

APA

Mude, R. G., Betta, A., & Debbarma, A. (2015). Capturing Node Resource Status and Classifying Workload for Map Reduce Resource Aware Scheduler. In Advances in Intelligent Systems and Computing (Vol. 309 AISC, pp. 247–257). Springer Verlag. https://doi.org/10.1007/978-81-322-2009-1_29

Register to see more suggestions

Mendeley helps you to discover research relevant for your work.

Already have an account?

Save time finding and organizing research with Mendeley

Sign up for free