Octopus: A multi-job scheduler for Graphlab

Abstract

Graphlab, which is a framework for large graph processing currently does not support multiple job scheduling simultaneously. However, for efficient use of the cluster resources, it may be required to share the cluster among multiple jobs. The challenges in multi-job scheduling in the case of graph processing are different from other frameworks such as Hadoop. In Hadoop, it is possible to schedule multiple jobs by fairly allocating resources to the jobs. We show in this paper that such an approach does not provide optimal results in the case of graph processing. We propose Octopus, a fair multi-job scheduler for Graphlab. The scheduler uses two different algorithms viz., First Fit with round robin Filling (FFF) and First In First Out with round robin Filling (FIFOF) to schedule large jobs of a user. We compare the performance of both the algorithms on a 20-node cluster. Preliminary results show that non-preemptive time sharing approach among users exhibits significant gain in turnaround time when compared to spatial resource sharing.