Design and Implementation of a Routing Control Platform
Matthew Caesar
UC Berkeley
Donald Caldwell
AT&T Labs-Research
Nick Feamster
MIT
Jennifer Rexford
Princeton University
Aman Shaikh
AT&T Labs-Research
Jacobus van der Merwe
AT&T Labs-Research
Abstract
The routers in an Autonomous System (AS) must dis-
tribute the information they learn about how to reach ex-
ternal destinations. Unfortunately, today’s internal Bor-
der Gateway Protocol (iBGP) architectures have serious
problems: a “full mesh” iBGP configuration does not
scale to large networks and “route reflection” can in-
troduce problems such as protocol oscillations and per-
sistent loops. Instead, we argue that a Routing Con-
trol Platform (RCP) should collect information about ex-
ternal destinations and internal topology and select the
BGP routes for each router in an AS. RCP is a logically-
centralized platform, separate from the IP forwarding
plane, that performs route selection on behalf of routers
and communicates selected routes to the routers using
the unmodified iBGP protocol. RCP provides scalability
without sacrificing correctness. In this paper, we present
the design and implementation of an RCP prototype on
commodity hardware. Using traces of BGP and inter-
nal routing data from a Tier-1 backbone, we demonstrate
that RCP is fast and reliable enough to drive the BGP
routing decisions for a large network. We show that RCP
assigns routes correctly, even when the functionality is
replicated and distributed, and that networks using RCP
can expect comparable convergence delays to those us-
ing today’s iBGP architectures.
1 Introduction
The Border Gateway Protocol (BGP), the Internet’s in-
terdomain routing protocol, is prone to protocol oscil-
lation and forwarding loops, highly sensitive to topol-
ogy changes inside an Autonomous System (AS), and
difficult for operators to understand and manage. We
address these problems by introducing a Routing Con-
trol Platform (RCP) that computes the BGP routes for
each router in an AS based on complete routing informa-
tion and higher-level network engineering goals [1, 2].
This paper describes the design and implementation of
an RCP prototype that is fast and reliable enough to co-
ordinate routing for a large backbone network.
1.1 Route Distribution Inside an AS
The routers in a single AS exchange routes to external
destinations using a protocol called internal BGP (iBGP).
Small networks are typically configured as a “full mesh”
iBGP topology, with an iBGP session between each pair
of routers. However, a full-mesh configuration does not
scale because each router must: (i) have an iBGP ses-
sion with every other router, (ii) send BGP update mes-
sages to every other router, (iii) store a local copy of
the advertisements sent by each neighbor for each des-
tination prefix, and (iv) have a new iBGP session con-
figured whenever a new router is added to the network.
Although having a faster processor and more memory
on every router would support larger full-mesh config-
urations, the installed base of routers lags behind the
technology curve, and upgrading routers is costly. In
addition, BGP-speaking routers do not always degrade
gracefully when their resource limitations are reached;
for example, routers crashing or experiencing persistent
routing instability under such conditions have been re-
ported [3]. In this paper, we present the design, imple-
mentation, and evaluation of a solution that behaves like
a full-mesh iBGP configuration with much less overhead
and no changes to the installed base of routers.
To avoid the scaling problems of a full mesh, today’s
large networks typically configure iBGP as a hierarchy of
route reflectors [4]. A route reflector selects a single BGP
route for each destination prefix and advertises the route
to its clients. Adding a new router to the system simply
requires configuring iBGP sessions to the router’s route
reflector(s). Using route reflectors reduces the memory
and connection overhead on the routers, at the expense
of compromising the behavior of the underlying network.
In particular, a route reflector does not necessarily select
˜SDI 05: ?d Smosim o ˜t?o??d Sstms Dsig & Im?m?tatio?#SE˜I_ Associatio?
5

RCP
eBGP
iBGP
Phy
sica
l
Peer
ing
Figure 1: Routing Control Platform (RCP) in an AS
the same BGP route that its clients would have chosen
in a full-mesh configuration. Unfortunately, the routers
along a path through the AS may be assigned differ-
ent BGP routes from different route reflectors, leading
to inconsistencies [5]. These inconsistencies can cause
protocol oscillation [6, 7, 8] and persistent forwarding
loops [6]. To prevent these problems, operators must en-
sure that route reflectors and their clients have a consis-
tent view of the internal topology, which requires config-
uring a large number of routers as route reflectors. This
forces large backbone networks to have dozens of route
reflectors to reduce the likelihood of inconsistencies.
1.2 Routing Control Platform (RCP)
RCP provides both the intrinsic correctness of a full-
mesh iBGP configuration and the scalability benefits of
route reflectors. RCP selects BGP routes on behalf of the
routers in an AS using a complete view of the available
routes and IGP topology. As shown in Figure 1, RCP
has iBGP sessions with each of the routers; these ses-
sions allow RCP to learn BGP routes and to send each
router a routing decision for each destination prefix. Un-
like a route reflector, RCP may send a different BGP
route to each router. This flexibility allows RCP to as-
sign each router the route that it would have selected in
a full-mesh configuration, while making the number of
iBGP sessions at each router independent of the size of
the network. We envision that RCP may ultimately ex-
change interdomain routing information with neighbor-
ing domains, while still using iBGP to communicate with
its own routers. Using the RCP to exchange reachability
information across domains would enable the Internet’s
routing architecture to evolve [1].
To be a viable alternative to today’s iBGP solutions,
RCP must satisfy two main design goals: (i) consis-
tent assignment of routes even when the functionality is
replicated and distributed for reliability and (ii) fast re-
sponse to network events, such as link failures and exter-
nal BGP routing changes, even when computing routes
for a large number of destination prefixes and routers.
This paper demonstrates that RCP can be made fast and
reliable enough to supplant today’s iBGP architectures,
without requiring any changes to the implementation of
the legacy routers. After a brief overview of BGP rout-
ing in Section 2, Section 3 presents the RCP architec-
ture and describes how to compute consistent forward-
ing paths, without requiring any explicit coordination be-
tween the replicas. In Section 4, we describe a proto-
type implementation, built on commodity hardware, that
can compute and disseminate routing decisions for a net-
work with hundreds of routers. Section 5 demonstrates
the effectiveness of our prototype by replaying BGP and
OSPF messages from a large backbone network; we also
discuss the challenges of handling OSPF-induced BGP
routing changes and evaluate one potential solution. Sec-
tion 6 summarizes the contributions of the paper.
1.3 Related Work
We extend previous work on route monitoring [9, 10] by
building a system that also controls the BGP routing de-
cisions for a network. In addition, RCP relates to re-
cent work on router software [11, 12, 13], including the
proprietary systems used in today’s commercial routers;
in contrast to these efforts, RCP makes per-router rout-
ing decisions for an entire network, rather than a single
router. Our work relates to earlier work on applying rout-
ing policy at route servers at the exchange points [14],
to obviate the need for a full mesh of eBGP sessions;
in contrast, RCP focuses on improving the scalability
and correctness of distributing and selecting BGP routes
within a single AS. The techniques used by the RCP for
efficient storage of the per-router routes are similar to
those employed in route-server implementations [15].
Previous work has proposed changes to iBGP that pre-
vent oscillations [16, 7]; unlike RCP, these other pro-
posals require significant modifications to BGP-speaking
routers. RCP’s logic for determining the BGP routes for
each router relates to previous research on network-wide
routing models for traffic engineering [17, 18]; RCP fo-
cuses on real-time control of BGP routes rather than
modeling the BGP routes in today’s routing system. Pre-
vious work has highlighted the need for a system that
has network-wide control of BGP routing [1, 2]; in this
paper, we present the design, implementation, and eval-
uation of such a system. For an overview of architec-
ture and standards activities on separating routing from
routers, see the related work discussions in [1, 2].
2 Interoperating With Existing Routers
This section presents an overview of BGP routing inside
an AS and highlights the implications on how RCP must
work to avoid requiring changes to the installed base of
IP routers.
˜SDI 05: ?d Smosim o ˜t?o??d Sstms Dsig & Im?m?tatio? #SE˜I_ Associatio?
;