Airavata Metascheduler: A Reliable, Fault Tolerant, and Resource-Aware Job Scheduling Service

Abstract

Software-as-a-service science gateways provide user interfaces and middleware for accessing scientific software deployed on remote high-performance computing resources and clusters. Selecting the resource to use for a particular job submission may be left to the user, who may need more information to make good choices when selecting from multiple options. To address this problem, we have designed and developed an extensible, scalable metascheduling system that can provide automated scheduling capabilities based on resource availability and other characteristics. We develop a system model based on queuing theory to guide our implementation and provide a basis for analysis. In particular, we derive an efficiency metric from these considerations. We implement the metascheduling system within the open-source Apache Airavata framework for science gateways as a supplemental service for guiding the job submission capabilities. We measure efficiency in representative scenarios, observing efficiencies of greater than 70% even in scenarios with high input rates and low job acceptance rates.