USENIX Association NSDI ’09: 6th USENIX Symposium on Networked Systems Design and Implementation 59
PADS: A Policy Architecture for Distributed Storage Systems
Nalini Belaramani
∗
, Jiandan Zheng
§
, Amol Nayate
†
, Robert Soule´
‡
,
Mike Dahlin
∗
, Robert Grimm
‡
∗
The University of Texas at Austin
§
Amazon.com Inc.
†
IBM TJ Watson Research
‡
New York University
Abstract
This paper presents PADS,apolicy architecture for build-
ing distributed storage systems. A policy architecture has
two aspects. First, a common set of mechanisms that al-
low new systems to be implemented simply by defining
new policies. Second, a structure for how policies, them-
selves, should be specified. In the case of distributed
storage systems, PADS defines a data plane that pro-
vides a fixed set of mechanisms for storing and trans-
mitting data and maintaining consistency information.
PADS requires a designer to define a control plane pol-
icy that specifies the system-specific policy for orches-
trating flows of data among nodes. PADS then divides
control plane policy into two parts: routing policy and
blocking policy. The PADS prototype defines a concise
interface between the data and control planes, it provides
a declarative language for specifying routing policy, and
it defines a simple interface for specifying blocking pol-
icy. We find that PADS greatly reduces the effort to de-
sign, implement, and modify distributed storage systems.
In particular, by using PADS we were able to quickly
construct a dozen significant distributed storage systems
spanning a large portion of the design space using just a
few dozen policy rules to define each system.
1 Introduction
Our goal is to make it easy for system designers to con-
struct new distributed storage systems. Distributed stor-
age systems need to deal with a wide range of hetero-
geneity in terms of devices with diverse capabilities (e.g.,
phones, set-top-boxes, laptops, servers), workloads (e.g.,
streaming media, interactive web services, private stor-
age, widespread sharing, demand caching, preloading),
connectivity (e.g., wired, wireless, disruption tolerant),
and environments (e.g., mobile networks, wide area net-
works, developing regions). To cope with these varying
demands, new systems are developed [12, 14, 19, 21,
22, 30], each making design choices that balance perfor-
mance, resource usage, consistency, and availability. Be-
cause these tradeoffs are fundamental [7, 16, 34], we do
not expect the emergence of a single “hero” distributed
storage system to serve all situations and end the need
for new systems.
This paper presents PADS,apolicy architecture that
simplifies the development of distributed storage sys-
tems. A policy architecture has two aspects.
First, a policy architecture defines a common set of
mechanisms and allows new systems to be implemented
simply by defining new policies. PADS casts its mech-
anisms as part of a data plane and policies as part of a
control plane. The data plane encapsulates a set of com-
mon mechanisms that handle the details of storing and
transmitting data and maintaining consistency informa-
tion. System designers then build storage systems by
specifying a control plane policy that orchestrates data
flows among nodes.
Second, a policy architecture defines a framework for
specifying policy. In PADS, we separate control plane
policy into routing and blocking policy.
• Routing policy: Many of the design choices of dis-
tributed storage systems are simply routing decisions
about data flows between nodes. These decisions pro-
vide answers to questions such as: “When and where
to send updates?” or “Which node to contact on a
read miss?”, and they largely determine how a sys-
tem meets its performance, availability, and resource
consumption goals.
• Blocking policy: Blocking policy specifies predicates
for when nodes must block incoming updates or lo-
cal read/write requests to maintain system invariants.
Blocking is important for meeting consistency and
durability goals. For example, a policy might block
the completion of a write until the update reaches at
least 3 other nodes.
The PADS prototype is an instantiation of this archi-
tecture. It provides a concise interface between the con-
trol and data planes that is flexible, efficient, and yet sim-
ple. For routing policy, designers specify an event-driven
program over an API comprising a set of actions that set
up data flows, a set of triggers that expose local node in-
formation, and the abstraction of stored events that store
and retrieve persistent state. To facilitate the specifi-
cation of event-driven routing, the prototype defines a
domain-specific language that allows routing policy to
be written as a set of declarative rules. For defining a
control plane’s blocking policy, PADS defines five block-
ing points in the data plane’s processing of read, write,

60 NSDI ’09: 6th USENIX Symposium on Networked Systems Design and Implementation USENIX Association
Simple
Client
Server
Full
Client
Server
Coda
[14]
Coda
+Coop
Cache
TRIP
[20]
TRIP
+Hier
Tier
Store
[6]
Tier
Store
+CC
Chain
Repl
[32]
Bayou
[23]
Bayou
+Small
Dev
Pangaea
[26]
Routing Rules 21 43 31 44 6 6 14 29 75 9 9 75
Blocking 5 6 5 5 3 3 1 1 4 3 3 1
Conditions
Topology Client/ Client/ Client/ Client/ Client/ Tree Tree Tree Chains Ad- Ad- Ad-
Server Server Server Server Server Hoc Hoc Hoc
Replication Partial Partial Partial Partial Full Full Partial Partial Full Full Partial Partial
Demand
√ √ √ √ √ √ √
caching
Prefetching
√ √ √ √ √ √ √ √ √ √
Cooperative
√ √ √ √ √ √
caching
Consistency Sequen- Sequen- Open/ Open/ Sequen- Sequen- Mono. Mono. Linear- Causal Mono. Mono.
tial tial Close Close tial tial Reads Reads izablity Reads Reads
Callbacks
√ √ √ √ √ √
Leases
√ √ √
Inval vs. Invali- Invali- Invali- Invali- Invali- Invali- Update Update Update Update Update Update
whole update dation dation dation dation dation dation
propagation
Disconnected
√ √ √ √ √ √ √ √ √
operation
Crash
√ √ √ √ √ √ √ √ √ √ √ √
recovery
Object store
√ √ √ √ √ √ √ √ √ √ √ √
interface*
File system
√ √ √ √ √ √ √ √ √ √ √
interface*
Fig. 1: Features covered by case-study systems. Each column corresponds to a system implemented on PADS, and the rows list
the set of features covered by the implementation.
∗
Note that the original implementations of some systems provide interfaces that
differ from the object store or file system interfaces we provide in our prototypes.
and receive-update actions; at each blocking point, a de-
signer specifies blocking predicates that indicate when
the processing of these actions must block.
Ultimately, the evidence for PADS’s usefulness is sim-
ple: two students used PADS to construct a dozen dis-
tributed storage systems summarized in Figure 1 in a few
months. PADS’s ability to support these systems (1) pro-
vides evidence supporting our high-level approach and
(2) suggests that the specific APIs of our PADS prototype
adequately capture the key abstractions for building dis-
tributed storage systems. Notably, in contrast with the
thousands of lines of code it typically takes to construct
such a system using standard practice, given the PADS
prototype it requires just 6-75 routing rules and a hand-
ful of blocking conditions to define each new systemwith
PADS.
Similarly, we find it easy to add significant new
features to PADS systems. For example, we add co-
operative caching [5] to Coda by adding 13 rules.
This flexibility comes at a modest cost to absolute per-
formance. Microbenchmark performance of an imple-
mentation of one system (P-Coda) built on our user-level
Java PADS prototype is within ten to fifty percent of the
original system (Coda [14]) in most cases and 3.3 times
worse in the worst case we measured.
A key issue in interpreting Figure 1 is understanding
how complete or realistic these PADS implementations
are. The PADS implementations are not bug-compatible
recreations of every detail of the original systems, but we
believe they do capture the overall architecture of these
designs by storing approximately the same data on each
node, by sending approximately the same data across the
same network links, and by enforcing the same consis-
tency and durability semantics; we discuss our definition
of architectural equivalence in Section 6. We also note
that our PADS implementations are sufficiently complete
to run file system benchmarks and that they handle im-
portant and challenging real world details like configura-
tion files and crash recovery.
2 PADS overview
Separating mechanism from policy is an old idea. As
Figure 2 illustrates, PADS does so by defining a data
plane that embodies the basic mechanisms needed for
storing data, sending and receiving data, and maintain-
ing consistency information. PADS then casts policy
as defining a control plane that orchestrates data flow
among nodes. This division is useful because it allows
the designer to focus on high-level specification of con-
trol plane policy rather than on implementation of low-
level data storage, bookkeeping, and transmission de-
tails.
PADS must therefore specify an interface between the
data plane and the control plane that is flexible and effi-
cient so that it can accommodate a wide design space. At
the same time, the interface must be simple so that the
designer can reason about it. Section 3 and Section 4 de-
tail the interface exposed by the data plane mechanisms
to the control plane policy.