Coordinating learning agents for multiple resource job scheduling

Abstract

Efficient management of large-scale job processing systems is a challenging problem, particularly in the presence of multi-users and dynamically changing system conditions. In addition, many real world systems require the processing of multi-resource jobs where centralized coordination may be difficult. Most conventional algorithms, such as load balancing, are designed for centralized, single resource problems. Indeed, in such a case, load balancing is known to provide optimal solutions. However, load balancing is not well suited to the more general, distributed, multi-resource allocation problem across heterogeneous networks that is frequently encountered in real world applications. Approaches based on heuristics can be designed to handle multi-resource allocation, but such approaches do not necessarily attempt to optimize directly a system-wide objective function. In this paper, we investigate a multiagent coordination approach to distributed, multi-resource job scheduling across heterogeneous servers. In this approach, agents at servers make local decisions to optimize an agent specific objective. The agent objectives though, are derived so that they are aligned with the overall efficiency of the system. We demonstrate that such a system outperforms (sometimes dramatically) more crudely constructed multiagent systems as well as a multi-resource version of load balancing. © 2010 Springer.