Krill: A compiler and runtime system for concurrent graph processing

Abstract

As a large number of emerging graph applications spread across different domains, the need for processing massive concurrent graph jobs (CGJs) is increasing. However, existing graph processing systems designed for a single job cannot efficiently tackle multiple CGJs, where they suffer from interfering memory access patterns and inefficient property management. In this paper, we introduce Krill, a compiler and runtime system for processing concurrent graph jobs. We propose an SAP model, which decouples graph structure, algorithm, and property. In the compiler, we propose leveraging the property buffer to easily write and manage property data. In the runtime system, we propose a novel technique named graph kernel fusion to reduce memory accesses, which fuses all the jobs and processes them as a whole. Experimental results show our system significantly reduces the number of memory accesses for CGJs by more than 6x compared with the baseline, and achieves up to 6.76x speedup with 3.84x shorter response latency than GraphM, the state-of-The-Art concurrent graph processing system.