Implementing Declarative Overlays
Boon Thau Loo
∗
UC Berkeley
Petros Maniatis
Intel Research Berkeley
Tyson Condie
∗
UC Berkeley
Timothy Roscoe
Intel Research Berkeley
Joseph M. Hellerstein
Intel Research Berkeley
UC Berkeley
Ion Stoica
UC Berkeley
ABSTRACT
Overlay networks are used today in a variety of distributed
systems ranging from file-sharing and storage systems to
communication infrastructures. However, designing, build-
ing and adapting these overlays to the intended application
and the target environment is a difficult and time consuming
process.
To ease the development and the deployment of such over-
lay networks we have implemented P2, a system that uses a
declarative logic language to express overlay networks in a
highly compact and reusable form. P2 can express a Narada-
style mesh network in 16 rules, and the Chord structured
overlay in only 47 rules. P2 directly parses and executes such
specifications using a dataflow architecture to construct and
maintain overlay networks. We describe the P2 approach,
how our implementation works, and show by experiment its
promising trade-off point between specification complexity
and performance.
Categories and Subject Descriptors
C.2.4 [Computer Communication Networks]: Distributed
Systems—distributed applications ; D.4.7 [Operating Sys-
tems]: Organization and Design—Distributed systems ; C.2.2
[Computer Communication Networks]: Network Pro-
tocols—protocol architecture, routing protocols
General Terms
Design, Experimentation, Languages
Keywords
Declarative overlays, dataflow engines, executable pseudocode
∗
Boon Thau Loo and Tyson Condie are supported in part by
the National Science Foundation under Grants No. 0205647,
0209108, and 0225660, and by a gift from Microsoft Corpo-
ration.
Appears in the 20th ACM Symposium on Operating Systems Principles,
Brighton UK, October 2005.
1. INTRODUCTION
Large-scale distributed systems inherently use one or more
application-level overlay networks as part of their operation.
In some cases, the overlay is prominent: for example, file-
sharing networks maintain neighbor tables to route queries.
In other systems the overlay or overlays may not be as ex-
plicit: for example, Microsoft Exchange email servers within
an enterprise maintain an overlay network among themselves
using a link-state algorithm over TCP for routing mail and
status messages.
This paper describes P2, a facility (deployable as a service
or library) for the declarative construction, maintenance,
and sharing of overlay networks. Applications submit to P2
a concise logical description of an overlay network, and P2
executes this to maintain routing data structures, perform
resource discovery, and optionally provide forwarding for the
overlay.
P2 is intended to greatly simplify the process of selecting,
implementing, deploying and evolving an overlay network
design. It is novel in (a) using a declarative logic language
for specifying overlays, and (b) employing a dataflow frame-
work at runtime for maintaining overlays instead of the more
conventional finite state machine approach. P2 automat-
ically compiles the declarative specification of the overlay
into a dataflow program, and can compile multiple overlay
specifications into a single dataflow.
We believe that these innovations together promise two
advantages: ease of specification, and sharing/reuse of code.
P2 overlay descriptions can be extremely concise. For exam-
ple, Chord [34] can be specified in 47 simple logic rules, ver-
sus thousands of lines of code for the MIT Chord reference
implementation and more than 320 statements in MACE-
DON [30], which is a much less complete implementation
than ours. Also, the high-level, declarative nature of P2
specifications means that they decompose cleanly into log-
ically reusable units: for example, a Symphony DHT [23]
might share many of the definitions in the Chord specifica-
tion.
This facilitates not only code reuse among systems, but
also the comparison, extension, and hybridization of overlay
designs within a single system. Moreover, describing over-
lays declaratively (effectively as queries) enables the natural
integration of distributed information-gathering tasks like
resource discovery and network status monitoring.
Unlike some other proposals for overlay toolkits, P2 does
not aim for performance results as good as optimized C,
C++, or Java implementations. Instead, our first aim is
to demonstrate that declarative overlay descriptions can be
1

implemented by P2 with acceptable performance, and that
there are benefits to the declarative specification that go
beyond the raw performance of a single overlay design. We
believe that this is useful for rapidly prototyping new ideas,
and eventually for deploying production systems as well.
This is not to say that P2 code is slow. P2’s memory
footprint running a full Chord implementation is relatively
small (about 800 kB of working set) and its CPU usage
is comparable to C++ implementations. However, the P2
specifications we discuss in this paper support a new design
point on the trade-off between a high degree of specification
compactness and the fine-grained timer tuning and adaptiv-
ity optimizations that pepper the code of mature, efficient
but painstaking overlay implementations.
Ultimately, our argument for P2 is similar to the argu-
ment for SQL and relational database management systems
some 35 years ago. The initial goals of our implementation
are also akin to those of the early relational database sys-
tems: to explore the feasibility of the declarative approach
in practice at a coarse grain, without trying to capture all
possible optimizations in the first generation of the system.
1.1 Contributions and Overview
This paper makes the following contributions. First, we
show how a diverse set of overlays can be expressed concisely
in a declarative specification language. Second, we show
how such specifications can usefully be executed as overlay
maintenance protocols – sharing communication, state, and
computation – by our implementation, the P2 distributed
dataflow engine. Finally, we demonstrate experimentally
that such overlays have acceptable performance compared
to hand-coded implementations.
The rest of this paper is structured as follows. In Sec-
tion 2 we outline the main features of our approach: using
a declarative logic language to specify an overlay, and com-
piling it to an executable graph of dataflow elements. We
contrast this approach to the typical techniques from the
literature. In Section 3 we discuss our implementation of
P2 and the specific challenges we encountered, and then in
Section 4 we examine in detail a relatively complex overlay
(Chord [34]) as implemented over P2. Section 5 evaluates
the performance of this network, and shows it to be accept-
able despite the simplicity of the specification. Section 6
situates our work in the context of other language-based ap-
proaches and related research in data processing systems.
We conclude in Section 7.
2. APPROACH
In this section we provide a broad overview of the P2 ap-
proach to overlay specification and runtime execution. In
the past, overlay networks have typically been characterized
in one of two ways. The protocol-centric approach favored by
MACEDON [30] traces its roots to event languages [13, 35]
that specify overlay execution via automata for event and
message handling. This style emphasizes the dynamics of
the overlay and its maintenance, but makes it difficult to de-
termine the overlay’s coarse structure and invariant proper-
ties. The alternative is a structure-centric approach, whose
roots can be traced to the specification of parallel inter-
connection networks [19]. This style, which has influenced
the literature on distributed hash tables (DHTs), specifies
overlays by focusing on a network graph structure (hyper-
cube, torus, de Bruijn graph, small-world graph, etc.), whose
invariant properties must be maintained via asynchronous
messaging. Unfortunately, graph-theoretic descriptions tend
to be expressed at a high level in natural language, and often
gloss over details of the actual runtime messaging. As a re-
sult, implementing structure-centric overlays often requires
a fair bit of engineering [10, 29], and different implemen-
tations of the same overlay can vary significantly in their
actual execution.
P2 spans the two approaches above, and expands upon
them in a way that we believe is particularly attractive for
overlay specification and runtime. The interface of P2 is
closer in spirit to the structure-centric approach, in that
it encourages the specification of overlays as logical struc-
tures with invariants. However, it also automatically com-
piles this specification to a dataflow program for managing
asynchronous messages, which looks closer to the protocol-
centric approach. We believe P2 improves upon previous
overlay specification work in either camp, by providing a
machine-interpretable description language based on rela-
tions among node states in the network, and by using a
dataflow runtime model instead of automaton-based proto-
cols.
Here, we provide a high-level view of the three compo-
nents of our approach: the use of relational tables to rep-
resent overlay state, our high-level declarative language to
specify the overlay’s logical properties and behavior, and
graphs of dataflow elements to represent runtime informa-
tion processing. The specific implementation details of these
components are deferred until Section 3.
2.1 Tables and Streams
We model an overlay as a distributed data structure, rep-
resented via a set of structured relations (sets of tuples) as
in a relational database. P2 employs two types of relations:
soft-state tables,andstreams of transient tuples, as in stream
query engines [4, 7, 26].
There are many ways to represent network graphs, but
the relational approach seems attractive for a variety of rea-
sons. First, structured tables are a simple and natural rep-
resentation for network state; for example, neighbor tables
are widely used in networks. Second, and more importantly
for our purposes, tables and relationships between them are
easy to represent concisely in a declarative language, as the
success of SQL has shown. Third, the distributed database
abstraction provides a consistently–named view of all the lo-
cal tables and messages at different nodes: queries and rules
can specify distributed state in a high-level, concise way.
Finally, the relational abstraction is a natural way to reuse
functionality and share routing state among different over-
lays. Tables with multiple indices can store tuples relevant
to several overlays or parts of overlays, which can select
elements from each table with their own criteria. For in-
stance, a table holding network links along with their mea-
sured capacity and latency can be shared between a latency-
conscious overlay as well as a capacity-conscious overlay. Ta-
ble names (with appropriate namespace scoping) provide a
natural way to share definitions between multiple overlay
specifications.
Our experience with overlay implementations has shown
that relations, together with some suitable mechanisms for
selecting tuples from each table, can fairly naturally repre-
sent the persistent routing state of the overlays we consid-
ered. We give examples later in support of this claim.
2