A Survey of Benchmarks for Graph-Processing Systems

Abstract

Benchmarking is a process that informs the public about the capabilities of systems-under-test, focuses on expected and unexpected system-bottlenecks, and promises to facilitate system tuning and new systems designs. In this chapter, we survey benchmarking approaches for graph-processing systems. First, we describe the main features of a benchmark for graph-processing systems. Then, we systematically survey across these features a diverse set of benchmarks for RDF databases, benchmarks for graph databases, benchmarks for parallel and distributed graph-processing systems, and data-only benchmarks. We trace in our survey not only the important benchmarks, but also their innovative approaches and how their core ideas evolved from previous benchmarking approaches. Last, we identify ongoing and future research directions for benchmarking initiatives.