An effective resource management in hadoop cluster using optimized algorithm

Abstract

Hadoop advances in executing the massive resources required by applications in a parallel and distributed computing environment, which uses the map-reduce framework to process the large dataset. In Hadoop we use two types of schedulers with YARN capabilities to run the application in big data environment namely Fair Scheduler and Capacitive Scheduler. Each scheduler has it is own queues and resource manager to allocate the resources to run the particular application. In this paper, introduction of PSO based centralized job queuing scheduler is used to manage and monitor the resources that will tune up the existing schedulers which gives the optimized resource utilization, speeds-up the execution and provides more active and dynamic execution of jobs in the big data environment.