Performance evaluation of statistical techniques for adaptive scheduling in autonomic systems

Abstract

The introduction of virtualization technology to grid and cloud computing infrastructures has enabled applications to be decoupled from the underlying hardware providing the benefits of portability, better control over execution environment and isolation. The virtualization layer also incurs a performance penalty, which can be significant for High Performance Computing (HPC) applications with high work volumes. Virtualization thus brings new requirements for dynamic adaptation of the scheduling to realize the potential flexibility of faster re-tasking and reconfiguration of workloads. Often scheduling approaches are based on some well-defined system-wide performance metric within the context of the given systems scope of operation. However, this is not optimized for the structure and behavior of specific applications having a mix of task types each with their own task precedences and resource requirements. Our work is concerned with combining virtualization and adaptive scheduling techniques to achieve an optimal balance between task placement flexibility and processing performance. In the scientific computing target systems, deadlines are attributed to tasks to ensure high throughput. We focus on application-specific dynamic adjustment of these deadlines to offset virtualization overhead. This paper reports our investigation of the performance of two adaptive scheduling algorithms, which borrow concepts from signal processing and statistical techniques such as probability distribution function. Job success and failure rate results from real task data from CERNs Large Hadron Collider Computer Grid (LCG) are also presented in this paper. © 2012 Springer-Verlag.