On global resource allocation in clusters for data analytics

Abstract

Hadoop YARN is one of the most commonly used frameworks for implementing MapReduce distributed computing model. The current resource allocation modes in YARN are triggered by events, which are executed when every slave sent heartbeat message to the master. In another word, the resource allocation is based on the order of every slave node, rather than the global information. A global resource allocation can achieve a better outcome than the allocation method based on every single node. In reality, resource allocation is a complicated issue and many influencing factors need to be considered. Based on the YARNs existing cluster architecture and allocation mode, this paper designs the mechanism of resource allocation and carries out work schedules to optimize the running time of cluster mainly focuses on network bandwidth and node execution rate. We make an improvement on the basis of the existing algorithm, and propose an algorithm used strategy based on the greedy choice to make resource allocation. We designed an experimental simulation of the operation of the clusters. Compared to the existing resource allocation model, the result shows our algorithm has improved the performance and shortens the execution time for the whole cluster.