How to Lease the Internet in Your Spare Time∗
Nick Feamster Lixin Gao Jennifer Rexford
Georgia Tech University of Massachusetts Princeton University
feamster@cc.gatech.edu lgao@ecs.umass.edu jrex@cs.princeton.edu
ABSTRACT
Today’s Internet Service Providers (ISPs) serve two roles:
managing their network infrastructure and providing (ar-
guably limited) services to end users. We argue that cou-
pling these roles impedes the deployment of new protocols
and architectures, and that the future Internet should support
two separate entities: infrastructure providers (who manage
the physical infrastructure) and service providers (who de-
ploy network protocols and offer end-to-end services). We
present a high-level design for Cabo, an architecture that en-
ables this separation; we also describe challenges associated
with realizing this architecture.
Categories and Subject Descriptors
C.2.1 [Computer Communication Networks]: Network
Architecture and Design
General Terms
Design, Management
1. Introduction
The Internet is relatively resistant to fundamental change.
The last fifteen years have offered countless “false starts”
in the deployment of new services. For example, differen-
tiated services, IP multicast, and secure routing protocols
have not seen wide-scale deployment, despite offering tangi-
ble value and making significant headway through the proto-
col standardization process. A major impediment to deploy-
ing these services is the need for coordination: an Internet
service provider (ISP) that deploys the service garners lit-
tle benefit until other domains follow suit. For example, an
ISP that deploys a secure routing protocol like S-BGP incurs
substantial cost but still is not protected from bogus route an-
nouncements unless other ISPs also deploy S-BGP.
ISPs are under immense pressure to offer “value added”
services, in response to both customer demands and the in-
creasing commoditization of Internet connectivity. Build-
ing a network with global reach requires either “building
it yourself” or relying on other ISPs for connectivity; ISPs
naturally adopt the latter approach to contain cost. Unfortu-
nately, because a single ISP rarely controls the entire path,
new services either have been deployed only in small islands
∗Apologies to Staniford et al. [11]. Note: This article is an editorial note
submitted to CCR. It has not been peer reviewed. Authors take full respon-
sibility for this article’s technical content. Comments can be posted through
CCR Online.
or have languished entirely. Some ISPs, hard-pressed to of-
fer profitable services, are driven to extortionary measures
such as degrading service for some, while providing “better
service” (though not better end-to-end service) for others, as
evidenced by the ongoing “net neutrality” debate.
Researchers are also under pressure to justify their work
in the context of a federated network by explaining how
new protocols could be deployed one network at a time, but
emphasizing incremental deployability does not necessarily
lead to the best architecture. In fact, focusing on incremen-
tal deployment may lead to solutions where each step along
the path makes sense, but the end state is wrong. Rather, we
argue that substantive improvements to the Internet architec-
ture may require fundamental change that is not incremen-
tally deployable. Unfortunately, ideas that are not incremen-
tally deployable are typically relegated to the library of pa-
per designs that are either never seen again, or, in rare cases,
dusted off as “band aid” fixes only when crisis is imminent
(as with IPv6 in the face of address depletion in IPv4).
We argue that decoupling infrastructure providers (who
deploy and maintain network equipment) from service
providers (who deploy network protocols and offer end-
to-end services)1 is the key to breaking this stalemate.
We propose Cabo (“Concurrent Architectures are Better
than One”), which exploits virtualization to allow a service
provider to simultaneously run multiple end-to-end services
over equipment owned by different infrastructure providers.
Cabo extends network virtualization beyond its current use
for supporting shared experimental facilities, such as Plan-
etLab [2] and GENI [6]. Rather than simply serving as an
evaluation platform for selecting a single “winning” archi-
tecture, support for virtual networks itself should be the ar-
chitecture. Cabo’s design adopts the pluralist philosophy [1]
that advocates a flexible and extensible system that supports
multiple simultaneous network architectures. We take the
pluralist approach further by refactoring the economics of
the Internet to have service providers form business relation-
ships with one or more infrastructure providers to construct
virtual networks that offer end-to-end services.
Decoupling service providers from infrastructure
providers is consistent with the business models that have
resulted from the commercialization of the Internet, and
some ISPs are already pushing the trend toward decoupling
service from infrastructure. The early days of the commer-
cial Internet saw the rise of “carrier hotels”, which reduce
1Throughout the paper, we use the term “service provider” as an organiza-
tion that composes network services and protocols on top of physical in-
frastructure, and “Internet service provider” to refer to a status quo ISP.

Provider 2
Infrastructure
Service Provider 1
Service Provider 2
End Host B
Provider 1
Infrastructure
End Host A
Provider 3
Infrastructure
Figure 1: Cabo architecture.
the cost of interconnection between ISPs by locating the
physical equipment of many different ISPs in the same
building. Co-location amortizes the high fixed cost of
maintaining a physical footprint (e.g., racks, power sup-
plies, backup generators, switches, fiber, “hands and eyes”
support, etc.) by sharing capital and operational expenditure
across ISPs. In the same way that connectivity providers
share infrastructure like backup generators, Cabo allows
service providers to share physical network infrastructure.
Now, several companies are starting to separate service
from infrastructure to provide better service or lower costs.
FON, a Spanish ISP, acts as third-party broker for existing
wireless access points deployed by private households [4].
FON brokers Internet access using physical infrastructure
deployed by other parties. Packet Fabric amortizes the cost
of inter-exchange connectivity in a single city by allowing
ISPs at these exchanges exchanges to share access to the
same switch [5]. Cabo pushes this philosophy further,
by allowing service providers to offer a wide range of
end-to-end services and network architectures, not just
Internet access or single switch ports.
Whether a “horizontally stratified” Internet would ever
arise hinges on many important economic questions.2 We
focus, however, on the many technical hurdles that must be
surmounted to realize Cabo. (As we describe in Section 2.2,
many of Cabo’s benefits can be realized even when a sin-
gle ISP deploys Cabo for the purposes of running multi-
ple architectures in parallel on the same network infrastruc-
ture.) These technical challenges, which we describe in Sec-
tion 2.3 include discovering the available physical infrastruc-
ture, mapping virtual networks into the underlying network
topology, and accounting for and provisioning resources.
2. Concurrent Architectures Better Than One
In this section, we describe a high-level overview of Cabo,
along with its associated benefits and challenges.
2.1 Cabo Architecture
Cabo separates the notion of conventional ISPs into
two distinct entities: infrastructure providers and service
providers. An infrastructure provider owns and maintains
2Separating infrastructure providers from service providers has precedence
in other industries. For example, the airline industry has airports (infrastruc-
ture providers) that allocate certain gates or terminals to particular airlines
(service providers). The leasing of commercial planes is another example.
In the U.S., airlines (service providers) rent planes from financial-services
firms (infrastructure providers). Similarly, shopping malls are typically
owned by different companies (infrastructure providers) than the commer-
cial chains that lease space for their stores (service providers).
the network equipment (e.g., routers and links) that forms an
infrastructure network. A service provider establishes agree-
ments with one or more infrastructure providers for access
to a share of these router and link resources. Cabo facili-
tates sharing of physical resources by subdividing a physical
node (i.e., router) or link into many virtual nodes and vir-
tual links. A virtual node controls a subset of the underly-
ing node resources, with guarantees of isolation from other
virtual nodes running on the same machine. Similarly, a vir-
tual link is formed from a path through the infrastructure
network and includes a portion of the resources along the
path. Cabo can guarantee bandwidth or delay properties on
these links using schedulers that arbitrate access to shared
resources, such as CPU, memory, and bandwidth.
A virtual network consists of virtual nodes and links that
belong to the same service provider. For example, in Fig-
ure 1, service provider 1 has a virtual network using physi-
cal resources belonging to infrastructure providers 1 and 3 to
provide end-to-end services between end hosts A and B. An
end host may run virtual machines that connect to different
virtual networks, possibly run by different service providers,
over a physical connection to one infrastructure provider.
Service providers may install software (e.g., a customized
routing protocol) on their virtual components and may even
program the hardware (e.g., a customized packet-forwarding
algorithm implemented on a network processor). A single
service provider may have multiple virtual networks tailored
to specific services or topologies. For example, one vir-
tual network may run an Interior Gateway Protocol (IGP)
like OSPF and conventional longest-prefix match packet for-
warding, while another virtual network may support source
routing based on flat addresses.
One service provider may offer service to another via
“nesting” of virtual components. That is, a virtual node
might even be subdivided into multiple virtual nodes, and a
virtual link itself comprises multiple virtual links. For exam-
ple, one service provider might provide end-to-end connec-
tivity (akin to an ISP today) and sell that connectivity to an-
other service provider that offers some other end-to-end ser-
vice. Also, an infrastructure provider might offer some ser-
vices beyond the basic support for virtual components. For
example, to reduce the number of nodes that other service
providers would need to manage, an infrastructure provider
might run a virtual network of its own, with virtual links be-
tween pairs of its edge routers.
2.2 The Benefits of Cabo
In this subsection, we present examples that illustrate the
benefits of Cabo, incluing easy deployment end-to-end net-
work services, the ability to run custom routing protocols,
and better accountability.
2.2.1 Better network services
End-to-end network services. Some players in the “net
neutrality” debate have advocated a tiered Internet, where
Internet service providers provide “better” service to edge
networks and content providers (e.g., Google) who pay more
money directly to those ISPs. This “enhanced service” is
disingenuous: a tiered Internet cannot inherently provide

better service, since no single ISP controls any given end-
to-end path (e.g., between a home user and Google). In
Cabo , service providers can add real value by exposing con-
trol over end-to-end paths. In Cabo, infrastructure providers
can achieve a competitive advantage by running more effi-
cient and robust networks, and service providers differenti-
ate themselves by running different end-to-end services on a
common physical infrastructure.
Customized protocols. Cabo allows service providers to
build virtual networks with dramatically different character-
istics on top of the same physical infrastructure. For ex-
ample, one service provider might deploy a virtual network
that provides strong security guarantees (e.g., by using a se-
cure routing protocol and self-certified addresses) at the ex-
pense of complete reachability, while another offers global
reachability and greater anonymity (e.g., by using a con-
ventional routing protocol with ephemeral IP addresses that
depend on a host’s current location). Similarly, one ser-
vice provider might perform conventional IP routing and for-
warding, while another permits end hosts to perform source
routing on a relatively small virtual network, consisting of
virtual links that span multiple hops in the infrastructure.
Deploying source routing today is immensely difficult, since
most ISPs disable the feature; in Cabo, a service provider
could decide to offer source routing on its virtual network
without having to coordinate with other ISPs.
Co-location for expanded network presence. In today’s
Internet, an organization that needs a global footprint must
deploy physical infrastructure in a wide variety of locations;
each router deployed in a new remote facility incurs a rel-
atively high fixed cost. Today, these organizations can con-
tract with an ISP that offers a Virtual Private Network (VPN)
service, though finding a single ISP with facilities at every
location may be difficult. In contrast, Cabo allows that enter-
prise (or its service provider) to instantiate virtual nodes and
links on equipment managed by an infrastructure provider
in diverse regions, allowing the organization to run its own
virtual network without incurring the costs of deploying and
managing its own physical equipment.
2.2.2 More robust management and operations
Testing and deploying new protocols. Today’s router soft-
ware is typically evaluated in a test lab before deployment.
Large lab configurations that mimic a production network
are expensive, and limiting tests to simple topologies and
traffic patterns may not give operators an accurate view of
how the new software would perform “in the wild”. In Cabo,
new router software (including new experimental services)
could be evaluated on a separate virtual network on the same
underlying infrastructure; this virtual network could initially
carry only test traffic or support users willing to serve as
early adopters. Also, migrating a network from one proto-
col to another can be painstaking. In Cabo, a new protocol
could be deployed in its own virtual network, followed later
by a cut-over of the data traffic from the old virtual network
to the new one.
Protection against misconfiguration. Cabo provides iso-
lation between different network components and services,
which can provide protection against misconfigurations and
bugs. Network protocols are commonly misconfigured and
are subject to implementation bugs. Adding a new service,
provisioning a new customer, or rebalancing traffic each
requires an operator either to invoke certain configuration
commands or to install new software; these actions may
cause instability or temporary service disruptions. Cabo al-
lows services that might interact to be compartmentalized
into different virtual networks, thereby preventing configu-
ration errors or software bugs related to one network service
from interfering with others.
Accountability at every layer. In the current Internet, a
single ISP manages its network from the physical infrastruc-
ture, all the way up to applications, but that ISP typically
does not have purview over an entire end-to-end path. When
performance or security problems arise, the ISP must initiate
the arduous process of locating the source of the fault (of-
ten a different ISP) and coordinating to diagnose and fix the
problem. This process is inherently difficult because both
monitoring and mitigation require coordination across one
or more administrative boundaries. Cabo, on the other hand,
allows each entity to have complete, end-to-end control over
the layer it is managing. For example, when the virtual com-
ponents do not behave as expected, the service provider has
direct recourse (and a direct business relationship) with the
infrastructure provider managing the equipment.
2.3 Challenges in Building Cabo
Realizing Cabo introduces many challenges that we are
exploring in our ongoing work. First, although Cabo al-
lows virtual networks to run customized protocols, we must
demonstrate that the underlying equipment can provide this
flexibility at high enough speed. In recent years, the major
router vendors have started supporting virtual routers to sim-
plify network designs, reduce capital expenditure, and lower
the barriers to co-location [9]. To better support new proto-
cols and forwarding algorithms, Cabo could also make use
of programmable routers [7, 13], which can be simultane-
ously used by multiple parties. Conventional techniques for
packet and CPU scheduling can ensure appropriate isolation
between virtual networks that run on the same infrastructure.
We must explore whether managing multiple specialized
virtual networks is less complex than managing one large
general purpose network. In particular, Cabo must support
provisioning and embedding, so that a service provider that
specifies a virtual topology, traffic demand matrix, availabil-
ity requirements, or some combination of these criteria can
be allocated an appropriate virtual network. Cabo also in-
troduces many management subtleties when physical com-
ponents fail. First, the Cabo substrate must notify virtual
network components when a failure or other disruption oc-
curs. Second, it must account for performance on each phys-
ical and virtual component to enable a service provider to
identify the cause of a performance or reliability problem in
the virtual network in the underlying physical infrastructure.
This approach to accountability is simpler than today’s situ-
ation, where an ISP many hops away may disrupt end-to-end
performance, without having any direct accountability to the
senders or receivers of traffic.

In Cabo, a service provider coordinates with one or
more infrastructure providers to create virtual networks,
which should be easier than coordinating across ISPs to de-
ploy new protocols and services today. Such coordination
might be achieved with a signaling protocol that allows ser-
vice providers to request virtual components; infrastructure
providers could apply admission control and embedding al-
gorithms to satisfy these requests. Requesting new virtual
networks requires service providers to communicate with the
infrastructure providers. To resolve this circularity, we be-
lieve that Cabo would ultimately have some virtual networks
that provide global reachability, much like today’s Internet,
which provides complete connectivity even though the net-
work does not intrinsically provide any such guarantee.
3. Related Work
Today’s Internet offers several examples of refactoring
connectivity to create new services. Equinix and Internap
allow edge networks to change upstream providers on rel-
atively short timescales, but these services can only con-
trol the first ISP along the path to the destination; in con-
trast, Cabo provides control over the entire end-to-end path.
Other systems allow hosts to request overlay paths with cer-
tain properties [8], without providing programmability in
the forwarding nodes. Content distribution networks and
bandwidth brokers also extend basic connectivity by creat-
ing paths from source to destination (or content). Cabo pro-
vides this functionality by making the construction of virtual
links a first-order primitive.
Cabo must allow many virtual networks to operate on the
same physical infrastructure. Cabo is similar to “switch-
lets”, which allow constructing virtual networks according
to some set of specified properties using subdivions of phys-
ical ATM switches and a standardized switch control inter-
face [14]. This architecture relied on the design of a common
interface to the switches but did not allow custom routing
software or forwarding engines run directly on the switches
themselves. Today’s layer-3 VPNs [10] also provide virtual-
ization functionality that is similar to Cabo. However, these
VPNs do not (in and of themselves) provide resource isola-
tion, they do not span multiple ISPs, and they offer service
providers neither access to the physical routers nor the abil-
ity to run custom routing software on these routers.
Some research infrastructures use virtualization to sup-
port multiple simultaneous experiments. PlanetLab [2] sup-
ports virtualization of network servers, but not complete net-
works. GENI [6] and VINI [3] focus on network virtualiza-
tion and programmability, with the goal of supporting ex-
periments rather than acting as an architecture itself. In ad-
dition, Cabo must bootstrap its own communications to net-
work elements, rather than relying on the legacy Internet.
In supporting programmable routers, Cabo resembles ac-
tive networks. Previous research on active networking fo-
cused on issues with mobile code (and the resulting language
and security issues) and providing control to end users [12].
In contrast, Cabo provides service providers (rather than
users) with their own virtual networks, with a fairly general
programming environment on the virtual nodes. In fact, a
service provider could run an active-network architecture in
a a virtual network on Cabo.
4. Conclusion
Although we are focusing on solving the technical prob-
lems with Cabo, we recognize that Cabo requires funda-
mental changes to the Internet’s operation that might could
hinder deployment. For example, infrastructure providers
would have to provide service providers access to their in-
frastructure. We note that Cabo does not prevent a single en-
tity from acting as both an infrastructure provider and a ser-
vice provider, and Cabo could provide benefits even to single
ISPs. Limited cooperation is already compatible with incen-
tives today, such as establishing geographical footprints by
leasing a virtual router in other ISPs.
In searching for a single “right” future Internet architec-
ture, Cabo suggests that the right future network architec-
ture is not an end state comprised of a collection of address-
ing, routing, and forwarding paradigms, but rather a plat-
form that allows these functions to evolve as demands on
communication networks change. Indeed, the designers of
IP aimed for generality, recognizing that they could not pre-
dict what networked applications would ultimately run on
top of the network. Continual rapid advances in communica-
tion technologies and the sheer difficulty of predicting future
requirements suggest that the architecture itself should be
sufficiently general to enable support for network protocols,
services, and architectures that we cannot imagine today.
REFERENCES
[1] T. Anderson, L. Peterson, S. Shenker, and J. Turner. Overcoming the
Internet impasse through virtualization. IEEE Computer,
38(4):34–41, Apr. 2005.
[2] A. Bavier, M. Bowman, D. Culler, B. Chun, S. Karlin, S. Muir,
L. Peterson, T. Roscoe, T. Spalink, and M. Wawrzoniak. Operating
System Support for Planetary-Scale Network Services. In Proc.
Networked Systems Design and Implementation, Mar. 2004.
[3] A. Bavier, N. Feamster, M. Huang, L. Peterson, and J. Rexford. In
VINI Veritas: Realistic and controlled network experimentation. In
Proc. ACM SIGCOMM, Sept. 2006.
[4] FON: WiFi everywhere! http://en.fon.com/, 2006.
[5] Private communication with Avi Freedman, Oct. 2006.
[6] GENI: Global Environment for Network Innovations. http://
www.geni.net/.
[7] E. Kohler, R. Morris, B. Chen, J. Jannotti, and M. F. Kaashoek. The
Click modular router. ACM Transactions on Computer Systems,
18(3):263–297, Aug. 2000.
[8] K. Lakshminarayanan, I. Stoica, and S. Shenker. Routing as a
Service. Technical Report UCB-CS-04-1327, UC Berkeley, 2004.
[9] D. McPherson et al. Core Network Design and Vendor Prophecies. In
NANOG 25, June 2003.
[10] E. Rosen and Y. Rekhter. BGP/MPLS VPNs. Internet Engineering
Task Force, Mar. 1999. RFC 2547.
[11] S. Staniford, V. Paxson, and N. Weaver. How to 0wn the Internet in
your spare time. In Proc. 11th USENIX Security Symposium, Aug.
2002.
[12] D. L. Tennenhouse and D. J. Wetherall. Towards an active network
architecture. ACM Computer Communications Review, Apr. 1996.
[13] J. Turner. A proposed architecture for the GENI backbone platform.
In Proc. Architectures for Networking and Communications Systems,
Dec. 2006.
[14] J. van der Merwe, S. Rooney, I. Leslie, and S. Crosby. The Tempest -
A Practical Framework for Network Programmability. IEEE
Network, 12(3):20–28, May 1998.