G2: A Graph Processing System for Diagnosing Distributed Systems
Zhenyu Guo† Dong Zhou‡ Haoxiang Lin† Mao Yang†
Fan Long‡ Chaoqiang Deng§ Changshu Liu† Lidong Zhou†
†Microsoft Research Asia ‡Tsinghua University §Harbin Institute of Technology
ABSTRACT
G2 is a graph processing system for diagnosing dis-
tributed systems. It works on execution graphs that
model runtime events and their correlations in distributed
systems. In G2, a diagnosis process involves a series of
queries, expressed in a high-level declarative language
that supports both relational and graph-based operators.
Each query is compiled into a distributed execution. G2’s
execution engine supports both parallel relational data
processing and iterative graph traversal.
Execution graphs in G2 tend to have long paths and
are in structure distinctly different from other large-
scale graphs, such as social or web graphs. Tailored
for execution graphs and graph traversal operations on
those graphs, G2’s graph engine distinguishes itself by
embracing batched asynchronous iterations that allows
for better parallelism without barriers, and by enabling
partition-level states and aggregation.
We have applied G2 to diagnosis of distributed sys-
tems such as Berkeley DB, SCOPE/Dryad, and G2 itself
to validate its effectiveness. When co-deployed on a 60-
machine cluster, G2’s execution engine can handle exe-
cution graphs with millions of vertices and edges; for in-
stance, using a query in G2, we traverse, filter, and sum-
marize a 130 million-vertex graph into a 12 thousand-
vertex graph within 268 seconds on 60 machines. The
use of an asynchronous model and a partition-level in-
terface delivered a 66% reduction in response time when
applied to queries in our diagnosis tasks.
1 INTRODUCTION
Distributed applications in data centers are increasingly
important as they power large-scale web and cloud ser-
vices. Often, the execution of such an application in-
volves a large number of cooperating processes running
on different machines, spanning multiple software mod-
ules and layers, tolerating and recovering from various
machine failures and network disruptions. Increases in
both the scale and complexity of such systems have made
it difficult to understand and diagnose their runtime (mis-
)behavior.
Typical diagnosis tasks start with observing misbe-
havior or anomaly, navigating through runtime informa-
tion such as logs to find relevant information, and pro-
cessing the information to infer root causes. For example,
starting with a log entry with an error message, diagnosis
could find all relevant log entries to infer the root cause
for the error. As another example, given two similar jobs
that noticeably perform differently, diagnosis could ex-
tract related runtime information to identify major differ-
ences. Also, it might be difficult to spot problems from a
large number of low-level runtime events. A useful prac-
tice is to aggregate performance information at an appro-
priate layer, identify which aggregated component in that
layer is problematic, and then drill down into the next
layer of details in an iterative process.
Effective diagnosis depends heavily on the ability
to correlate runtime events and to leverage these corre-
lations. Previous work, especially those on path-based
analysis [14, 7, 13, 8, 19, 26, 18, 27], has largely ad-
dressed the important problem of generating and cor-
relating runtime information from executions of a dis-
tributed system. Often the difficulty for diagnosis is not
due to lack of information, but due to the inability to nav-
igate through and process a sea of information to find out
what is relevant.
In this paper, we propose G2, a distributed graph pro-
cessing system for storing runtime information of dis-
tributed systems and for processing queries on such in-
formation. Runtime information is organized as a graph,
where vertices correspond to events and edges corre-
spond to correlations between events. Diagnosis then in-
volves an iterative process of writing queries against the
graph and analyzing the results of those queries. G2 pro-
vides a declarative language that supports relational and
graph operators that operate on the graph structure. For
example, given an error log entry e, a G2 query can be is-
sued to find all events (vertices) that vertex e is causally
dependent on, where causal dependencies are captured
by certain types of edges. This query uses a slicing oper-
ator that G2 provides. From a starting vertex v, forward
slicing finds all vertices that causally and transitively de-
pend on v, while backward slicing finds all vertices that
v is dependent on.
Graph aggregation and summarization are another ef-
fective way of reducing the amount of information to be
examined during diagnosis. In an execution graph, each
vertex is associated with a context that indicates the ag-
gregation units that the event belongs to. Examples of ag-
gregation units include static ones such as components,
classes, and functions, as well as dynamic ones such as
machines, processes, and threads. A G2 query can ag-
gregate information at an appropriate level. For example,
to compare executions of two jobs, a query can compute

the forward slices from the starting points of two jobs. To
make comparison easier, the query can continue to com-
pute a machine-level aggregation from the two slices.
This requires a hierarchical aggregation graph operator
that transforms an input graph into a smaller one: it con-
denses each continuous segment of events with the same
aggregation unit (e.g., machine) to create a single super-
node and applies an aggregation function on those events
to compute the an associated aggregated value.
Distributed query execution in G2 is supported by a
distributed storage and execution system that addresses
the challenges of storing and processing large execution
graphs with millions or even billions of vertices effi-
ciently. In G2, events and correlations are captured on
local machines as they occur during system execution,
leading to a natural partitioning of an execution graph.
G2’s execution engine is tailored for execution graphs
that exhibit significantly different characteristics from
other large graphs, such as social and web graphs. Ex-
ecution graphs tend to have long paths corresponding to
events along a logically related progression of execution,
where social and web graphs have relatively small diam-
eters. Graph operations on execution graphs are often in
the form of graph traversal, which is again different from
iterative graph operations that must proceed in globally
synchronized rounds, such as in page-rank computation
for example. Consequently, G2 embraces batched asyn-
chronous iterations, where processing on each partition
is batched, but does not have to proceed synchronously
in lock steps. Both slicing and hierarchical aggregation
fall into this model that allows for improved parallelism
and efficiency than the bulk synchronous computation
model in previous work, such as in Pregel [24]. Barri-
ers are used only at the end of graph traversal or to create
global consistent checkpoints for failure recovery. Fur-
thermore, partitions tend to contain long local paths be-
fore those paths connect to vertices on other partitions
due to cross-machine communication. Graph traversal
within each partition is therefore significant to the overall
graph traversal performance. Instead of a vertex-oriented
interface, G2 exposes a partition-oriented interface that
allows partition-level aggregation states to be maintained
in an appropriate data structure. This is particularly valu-
able for hierarchical aggregation, where the choice of
partition-level data structure significantly influences per-
formance.
We have built a prototype and applied it to a
set of distributed systems, including Berkeley DB [2],
SCOPE/Dryad [11, 22], and G2 itself. Berkeley DB is a
replicated distributed key-value database that can be eas-
ily linked with applications. SCOPE/Dryad is a produc-
tion data intensive computation system, which includes
a distributed file system, a distributed execution engine
(Dryad), and a declarative query language (SCOPE). G2
is shown to be effective in diagnosis: for instance, us-
ing a query in G2, we traverse, filter, and summarize a
130 million-vertex graph into a 12 thousand-vertex graph
within 268 seconds on 60 machines. The optimizations
we introduce into G2’s execution engine are effective: the
use of asynchronous model and partition-level interface
delivered up to a factor of 3 performance improvement
when applied to graph operators in our diagnosis tasks.
We have also studied scalability of G2 and the check-
pointing overhead introduced to enable failure recovery.
The contribution of G2 is two-fold. First, as a tool,
G2 enables efficient distributed-system diagnosis by al-
lowing users to write declarative queries with both rela-
tional and graph operators, and by providing a distributed
engine that executes those queries efficiently. Second, as
a distributed system, G2’s execution engine targets a dif-
ferent type of graphs with different structural characteris-
tics and with different type of graph operations. It allows
a batched asynchronous graph computation model and a
partition-level interface, which have contributed signifi-
cantly to its efficiency.
The rest of the paper is organized as follows. Sec-
tion 2 introduces the system execution graph data model,
and the diagnosis primitives applied to the graph. Sec-
tion 3 presents the operators, and the language that
G2 supports, as well as several examples expressed in
those constructs. The design and optimization of the dis-
tributed graph engine is the focus of Section 4, followed
by implementation details in Section 5. We evaluate G2
and share experience in Section 6. Section 7 discusses
the related work. Finally, we conclude in Section 8.
2 MODEL
Distributed-system diagnosis in G2 centers on the data
model and the operations defined on the model, which
are the topic of this section.
2.1 Text, Paths, and Graphs
Traditionally, system diagnosis treats runtime infor-
mation (e.g., logs) as unstructured text and involves a te-
dious and ineffective process of going through logs using
primitive text-processing tools such as grep. Using grep
on a special tag (such as a request id) captures all entries
that are explicitly related to that request, but is likely to
miss information that has implicit dependencies.
Previous work [14, 7, 13, 8, 20, 18, 27] on correlating
runtime information has effectively addressed this short-
coming by capturing common causal relationship in dis-
tributed systems. A path-like abstraction is often used to
track how a request flows through a distributed system.
This relatively simple structure is effective for request-
centric analysis and modeling, and reflects a good bal-
ance between what an abstraction enables, the simplicity
2