Runtime estimation and scheduling on parallel processing supercomputers via instance-based learning and swarm intelligence

Abstract

Supercomputing has been indispensable in the unstoppable trend of high-speed computing evolution. This work aims at improving its running efficacy by introducing a new two-step scheduling approach. Based on the analysis of large historical data, we provide an accurate runtime estimation scheme using Instance-Based Learning (IBL) in the first step. Then a swarm intelligence based scheduling (SIBS) method is proposed to optimize the scheduling performance in terms of total runtime makespan and fair resource allocation. A method comparison on a dataset from the ALPS supercomputer, which consists of 804k workload data in 2016, shows that our proposed method outperforms the most commonly used strategy-Extensible Argonne Scheduling System (EASY).