Guidelines for Interdomain Traf c Engineering
Nick Feamster Jay Borkenhagen Jennifer Rexford
Laboratory for Computer Science AT&T IP Services Internet and Networking Systems
Massachusetts Institute of Technology AT&T Labs AT&T Labs Research
Cambridge, MA 02139 Middletown, NJ 07748 Florham Park, NJ 07932
feamster@lcs.mit.edu jayb@att.com jrex@research.att.com
Abstract
Network operators must have control over the ow of traf c into,
out of, and across their networks. However, the Border Gateway
Protocol (BGP) does not facilitate common traf c engineering tasks,
such as balancing load across multiple links to a neighboring AS or
directing traf c to a different neighbor. Solving these problems is
dif cult because the number of possible changes to routing poli-
cies is too large to exhaustively test all possibilities, some changes
in routing policy can have an unpredictable effect on the ow of
traf c, and the BGP decision process implemented by router ven-
dors limits an operator’s control over path selection.
We propose fundamental objectives for interdomain traf c engi-
neering and speci c guidelines for achieving these objectives within
the context of BGP. Using routing and traf c data from the AT&T
backbone we show how certain BGP policy changes can move traf-
c in a predictable fashion, despite limited knowledge about the
routing policies in neighboring AS’s. Then, we show how opera-
tors can gain greater exibility by relaxing some steps in the BGP
decision process and ensuring that neighboring AS’s send consis-
tent advertisements at each peering location. Finally, we show that
an operator can manipulate traf c ef ciently by changing the routes
for a small number of pre xes (or groups of related pre xes) that
consistently receive a large amount of traf c.
Categories and Subject Descriptors
C.2.2 [Computer-Communication Networks]: Network Protocols
Routing Protocols; C.2.3 [Computer-Communication Networks]:
Network Operations Network management, Network monitoring,
Public networks; C.2.5 [Computer-Communication Networks]:
Local and Wide-Area Networks Internet
General Terms
Measurement, Performance
1. Introduction
Operating a large IP backbone requires continuous attention to
the distribution of traf c over the network. Equipment failures and
changes in routing policies in neighboring domains can trigger sud-
den shifts in the ow of traf c. Flash crowds caused by special
events and new applications can also cause signi cant changes in
the load on the network. Network failures and traf c uctuations
degrade user performance and lead to inef cient use of network re-
sources. Network operators adapt to changes in the distribution of
traf c by adjusting the con guration of the routing protocols run-
ning on their routers. Additionally, routing con guration changes
are often necessary after deploying new routers and links. Develop-
ing effective techniques for adapting routes to the prevailing traf c
and topology has been an active area of research and standards ac-
tivity during recent years [1, 2, 3, 4]. Previous work on traf c engi-
neering has focused predominantly on Interior Gateway Protocols
(IGPs), such as OSPF, IS-IS, and MPLS, which control the ow of
traf c within a single Autonomous System (AS).
In practice, though, most traf c in a large backbone network tra-
verses multiple domains, making interdomain routing an important
part of traf c engineering. We motivate the need for interdomain
traf c engineering with three examples:
Congested edge link: The links between domains are com-
mon points of congestion in the Internet. Upon detecting an
overloaded edge link, an operator can change the interdo-
main paths to direct some of the traf c to a less congested
link.
Upgraded link capacity: Operators of large IP backbones
frequently install new, higher-bandwidth links between do-
mains. Exploiting the additional capacity may require rout-
ing changes that divert traf c traveling via other edge links
to the new link.
Violation of peering agreement: An AS pair may have a busi-
ness arrangement that restricts the amount of traf c they ex-
change; for example, the outbound and inbound traf c may
have to stay within a factor of 1.5. If this ratio is exceeded,
an AS may need to direct some traf c to a different neighbor.
The state of the art for interdomain traf c engineering is ex-
tremely primitive. The IETF’s Traf c Engineering Working Group,
which has focused almost exclusively on intradomain traf c engi-
neering, recently noted that interdomain traf c engineering is usu-
ally applied in a trial-and-error fashion. A systematic approach for
inter-domain traf c engineering is yet to be devised [1]. Opera-
tors make manual changes in the routing policies without a good
understanding of the effects on the ow of traf c or the impact on
other domains.
Ultimately, this ad hoc approach to interdomain traf c engineer-
ing must evolve into mature, well-tested guidelines and mecha-
nisms. This paper is a rst step in that direction. Recent previous
work has presented a high-level overview of interdomain traf c en-
gineering [5] and described the traf c data that must be measured
to perform interdomain traf c engineering [6]. In addition, several
commercial products help large campus and corporate networks
balance load over connections to multiple upstream providers [7];
however, these products do not address the challenges of traf c en-
gineering for large ASes in the core of the Internet. Our work is
the rst to propose fundamental objectives for interdomain traf c
engineering, as well as speci c guidelines for service providers to
achieve these objectives within the context of BGP. We argue that

the guidelines we present are practical by characterizing traf c and
routing data from a large, tier-1 IP backbone.
Neighboring ASes use the Border Gateway Protocol (BGP) to
exchange routing information to provide end-to-end connectivity
between hosts in different domains [8, 9, 10]. Each BGP advertise-
ment announces reachability to a destination pre x that represents
a block of IP addresses. Each advertisement includes a list of the
ASes in the path, along with several other attributes. The routers
in each AS apply local routing policies that manipulate these at-
tributes to in uence the selection of the best route for each destina-
tion pre x and to decide whether to propagate this route to neigh-
boring ASes. Operators affect the ow of traf c by tuning the local
routing policies that affect the selection of the best path for a des-
tination pre x. Choosing the appropriate con guration is dif cult
since it depends on the network topology, the IGP parameters, the
BGP advertisements from neighboring ASes, and the current traf-
c patterns. Our work focuses on the impact of BGP policies on
the ow of traf c leaving an AS at the egress points that connect
to neighboring domains. Some traf c engineering tasks necessitate
changes to how traf c enters the network. However, controlling
how traf c enters the network in a predictable way requires coor-
dination with neighboring domains [1]. The results of our analysis
of outbound traf c can be applied by the neighboring ASes to in-
uence how traf c enters the network.
Interdomain traf c engineering is signi cantly more complicated
than intradomain traf c engineering. While IGPs select paths based
on link metrics, such as static weights or dynamic load informa-
tion, BGP advertisements do not explicitly convey any information
about the resources available on a path. BGP routing policies are
complex and depend on a variety of factors, such as the commer-
cial relationships with neighboring ASes [11]. The selection of
the best path for each pre x depends not only on the routing poli-
cies but also on the advertisements sent by neighboring domains.
Operators have, at best, indirect in uence on BGP path selection.
In fact, changing the BGP policy in one AS may alter the adver-
tisements propagated to neighboring domains, which may inadver-
tently affect how traf c enters the AS. The constraints that BGP
imposes on making good routing decisions makes moving to a
radically different interdomain routing paradigm desirable, but ex-
tremely dif cult in practice. Rather than proposing a new routing
protocol, our analysis identi es ways to support traf c engineering
within the existing BGP framework.
Router vendors support a wide variety of con guration com-
mands that provide signi cant exibility in specifying BGP poli-
cies. Selecting the right policy changes for a particular traf c-
engineering task is challenging, especially for service providers
that have many connections to neighboring domains. Our study
focuses on developing traf c engineering techniques that achieve
the following objectives:
Achieving predictable traf c ow changes: Some routing
changes have effects that are dif cult to predict in advance,
due to the routing policies in other domains. Our analysis
identi es approaches for tuning policies in ways that have
predictable outcomes and limit the changes seen by neigh-
boring domains.
Limiting the in uence of neighboring domains: Certain prac-
tices, such as sending inconsistent advertisements at differ-
ent peering locations, can have a signi cant impact on the
path selection process. Our analysis shows how operators
can check for these practices and use BGP policies that limit
their effects.
Reducing the overhead of routing changes: Changing the
routing policy may trigger new advertisements that impose
a load on the routers and a delay for converging to a new set
of routes. Our analysis shows that operators can limit over-
head by focusing on the small number of pre xes (or groups
of pre xes) that consistently receive a large amount of traf c.
Although this paper primarily describes how to achieve these ob-
jectives within the context of BGP, these objectives are applicable
to interdomain traf c engineering in general. We discuss our re-
sults for these three objectives after a brief background section on
the BGP protocol and traf c engineering tools and an overview of
our measurement data from AT&T’s IP backbone.
2. BGP Traf c Engineering
This section presents an overview of BGP and the attributes as-
sociated with route advertisements. We brie y describe tools that
could allow operators to adjust the routing con guration to the pre-
vailing traf c.
2.1 Border Gateway Protocol
Internet routing operates at the level of address blocks, or pre-
xes. Each pre x consists of a


-bit address and a mask length;
for example, 
  

 

represents the
 addresses ranging
from 

 
  to 

 

 . An IP router constructs a forward-
ing table that is used to select the output interface for each incom-
ing packet, based on the longest-matching pre x for that destina-
tion address. Routers in different ASes use BGP to exchange up-
date messages about how to reach different destination pre xes. A
router sends an announcement to notify its neighbor of a new route
to the destination pre x and sends a withdrawal to revoke the route
when it is no longer available. Each advertisement includes a num-
ber of attributes about the route, including the list of ASes along the
path to the destination pre x. Before accepting an advertisement,
the receiving router checks for the presence of its own AS number
in the AS path to detect and remove routing loops.
A router may receive routes for the same pre x from multiple
neighboring ASes. The router applies import policies to lter un-
wanted routes and to manipulate the attributes of the remaining
routes. Ultimately, the router invokes a decision process to se-
lect exactly one best route for each destination pre x among
all the routes it hears. The router then applies export policies to
manipulate attributes and decide whether to advertise the route to
neighboring ASes. In addition to exchanging BGP messages with
neighboring domains, an AS may use internal BGP (iBGP) to dis-
tribute routing information among its routers. Ultimately, every
router must select a single best route for each pre x among the
advertisements from the various external BGP (eBGP) and iBGP
neighbors.
BGP advertisements can include numerous attributes [9], and the
BGP decision process implemented by router vendors has several
steps, which proceed in order and sequentially eliminate candidates
for the best route [12, 13, 14]. To simplify the discussion, we focus
on ve main steps in the selection process:
1. Highest local preference: Prefer routes with the highest local
preference, assigned by the import policy and conveyed to
other routers via iBGP.
2. Shortest AS path: Prefer routes with the shortest AS path
length, as conveyed in the BGP advertisement.
3. eBGP over iBGP: Prefer routes learned via eBGP over routes
learned via iBGP, since leaving the AS directly is preferable
to traveling through the AS.