Hosting Virtual Networks on Commodity Hardware
Sapan Bhatia∗, Murtaza Motiwala†, Wolfgang M¤uhlbauer‡, Vytautas Valancius†,
Andy Bavier∗, Nick Feamster†, Larry Peterson∗, and Jennifer Rexford∗
∗ Princeton University † Georgia Tech ‡ T›Labs/TU Berlin
ABSTRACT
This paper describes Trellis, a software platform for hosting
multiple virtual networks on shared commodity hardware.
Trellis allows each virtual network to de ne its own topol-
ogy, control protocols, and forwarding tables, which low-
ers the barrier for deploying custom services on an isolated,
recon gurable, and programmable network, while amor-
tizing costs by sharing the physical infrastructure. Trellis
synthesizes two container-based virtualization technologies,
VServer and NetNS, as well as a new tunneling mechanism,
EGRE, into a coherent platform that enables high-speed vir-
tual networks. We describe the design and implementation,
of Trellis, including kernel-level performance optimizations,
and evaluate its supported packet-forwarding rates against
other virtualization technologies. We are in the process of
upgrading the VINI facility to use Trellis. We also plan to
release Trellis as part of MyVINI, a standalone software dis-
tribution that allows researchers and application developers
to deploy their own virtual network hosting platforms.
1. Introduction
Network services that have strikingly different require-
ments in terms of throughput, packet loss, security, or stabil-
ity must nevertheless operate over a single, common com-
munication infrastructure. Many of these services would
bene t from having their own network topologies, and direct
control over the routing, forwarding, and addressing mecha-
nisms. For example:
• Interactive applications (e.g., gaming, VoIP), which
could run an application-speci c routing protocol that
converges more quickly than the existing network pro-
tocols that favor scalability over fast convergence.
• Critical services, which could run on a separate net-
work with protocols tailored to defend against un-
wanted traf c ( e.g., denial-of-service attacks).
• Enterprises, which could construct and lease a
private network connecting geographically disparate
sites, with in-network support for key applications.
• Network service providers, which could run a sepa-
rate development network for deploying and testing
new con gurations, protocols, and designs, and sup-
porting early-adopter customers.
All of these needs could be addressed with the deployment
of separate physical networks, each with customized proto-
cols and topologies. Unfortunately, deploying physical in-
frastructure per service incurs tremendous space, power, and
management costs. Customizing today’s network devices is
also challenging (and often impossible) because many sup-
port only limited, proprietary interfaces. These needs could
also be addressed by deploying an overlay network that runs
customized software on a distributed collection of comput-
ers connected to the Internet. However, overlays have lim-
ited visibility into, and control over, network conditions, and
typically cannot forward traf c at high speeds.
Instead, we propose a network hosting platform that
can run multiple programmable virtual networks over a
shared physical network infrastructure. This hosting plat-
form should have the following properties:
• Speed: A virtual network should be able to forward
packets up to multi-Gigabit speeds.
• Isolation: To prevent virtual networks from interfer-
ing with one another, the infrastructure should support
namespace and resource isolation of system resources
(e.g., process IDs, les, CPU) as well as network re-
sources (e.g., forwarding tables, link bandwidth).
• Flexibility: A service running inside the virtual net-
work should be able to de ne its own routing proto-
col and application logic. The platform should provide
a powerful and familiar development environment for
network services.
• Scalability: The platform should be able to support
many such virtual networks simultaneously to amor-
tize its deployment and maintenance costs.
• Low cost: The cost for hosting a virtual network
should be very low. Our hosting system should run on
commodity hardware (i.e., server-class PCs) to reduce
costs and barriers to entry. Using commodity hardware
also allows the infrastructure to more cheaply track ad-
vances in new technology (e.g., multicore processors).
Our primary contribution is the design and implementa-
tion of Trellis, a platform for hosting virtual networks that
achieves these goals. Trellis provides the substrate on top
of which multiple fast and exible virtual networks can run.
Trellis synthesizes existing virtualization technologies (for
virtualizing hosts and network stacks) with a new tunnel-
ing protocol and a new fast software bridge kernel module,
to provide a scalable hosting platform with good isolation
between virtual networks. The key challenge was to iden-
tify the right combination of technologies that could best
satisfy our design goals: performance, scalability, isolation,
and exibility. We believe that the design choices and trade -
offs we have made hit a sweet spot along these axes for a
virtual network hosting platform.

Our experiments demonstrate that Trellis is fast and scal-
able and provides good isolation between virtual networks.
A virtual network hosted on Trellis can forward pack-
ets more than ten times faster than a similar overlay net-
work [10]. Nodes running Trellis can host more than 60
virtual networks in parallel with no noticeable degradation
in performance. The performance and jitter in any virtual
network is nearly identical to that seen on native hosts.
We are deploying the Trellis software on our wide-area
Virtual Network Infrastructure (VINI) facility [10]. The goal
of VINI is to enable researchers to evaluate new protocols
and deploy new services in an environment that is both real-
istic (e.g., runs real routing software and carries real traf c)
and controlled. We believe Trellis’s high performance and
low jitter are a step towards achieving these goals. Because
the need for low cost, isolated, recon gurable, and pro-
grammable networks extends beyond the applications that
will run on our modest-sized VINI facility, we also plan to
release the Trellis software as part of MyVINI, a software
distribution that allows researchers, network designers, and
application developers to deploy their own virtual network
hosting infrastructures. Beyond Trellis, MyVINI includes
software for instantiating the virtual networks, including al-
locating system resources such as CPU and bandwidth.
The rest of the paper is organized as follows. Section 2 de-
scribes Trellis’s design, its two constituent components (i.e.,
virtual hosts and virtual links), and their integration to sup-
port a virtual network. Section 3 assesses whether (and how)
existing virtualization technologies, often designed for a dif-
ferent purpose, can support virtual hosts and virtual links in
Trellis. Section 4 describes the implementation choices we
made for Trellis to ful ll our design goals. Section 5 evalu-
ates Trellis’s performance, scalability, and isolation relative
to both native packet forwarding and other virtualization al-
ternatives. Section 6 discusses our ongoing work, and Sec-
tion 7 concludes.
2. Virtual Networks on Commodity Hardware
A virtual network is built using two components:
1. virtual hosts, which run software and forward packets,
and
2. virtual links, which transport packets between virtual
hosts.
Virtual networks constructed using commercial routers that
support virtualization [24] currently have the limitation that
they can only run one speci c application (proprietary rout -
ing software and operating system) inside a virtual host. In
contrast, Trellis is a virtual-network substrate that can run on
commodity hardware using general-purpose operating sys-
tems. Trellis is likely to be cheaper to deploy than com-
mercial offerings and can support a wide range of network
services and applications. In this section we drive the design
of Trellis from our requirements for (1) the entire system,
and (2) the virtual hosts and virtual links that comprise it.
Figure 1: Overview of host virtualization, with virtual hosts shaded.
2.1 Trellis Design Requirements
We identify four high-level design requirements for Trel-
lis. First and foremost, it must connect virtual hosts with
virtual links to construct a virtual network. Second, it must
run on commodity hardware (i.e., server-class PCs) in or-
der to keep deployment, expansion, and upgrade costs low.
Third, it runs a general-purpose operating system inside the
virtual hosts that can support existing routing software (e.g.,
XORP [18] and Quagga [5]) as well as provide a convenient
and familiar platform for developing new services. Finally,
Trellis should support packet forwarding inside the kernel
of the general-purpose OS, since forwarding every packet in
user space introduces signi cant overhead and reduces the
packet forwarding rate. An application running in userspace
inside a virtual host can interact with devices representing
the end-points of virtual links, and can write forwarding
table entries (FTEs) to an in-kernel forwarding table (for-
warding information base, or FIB) to control how the ker-
nel forwards packets between the virtual links. Together, we
believe that these design requirements place Trellis in the
sweet spot mentioned earlier: it is the design point that
best satis es our overall goal of hosting fast, exible virt ual
networks on a scalable and low-cost platform.
Understanding the design of Trellis rst requires a better
understanding of the requirements for both virtual hosts and
virtual links. The next two sections de ne these components
and detail their requirements.
2.2 Virtual Hosts
A virtual host sees the illusion of a dedicated physical
host, even though multiple virtual hosts may be running on
the same physical hardware. At a high level, a virtual host
can be thought of as a box containing resources, as illus-
trated in Figure 1. A virtual host appears to have dedicated
physical or logical resources inside this box; examples of
physical resources are CPU, memory, and link bandwidth,
whereas logical resources are resources implemented by the
operating system such as the process table, page table, IPv4
forwarding table, memory buffers, etc. In reality, all of these
resources are only virtual in that they are enabled by a vir -
tualization layer that implements the virtual host abstraction.
The virtualization layer creates virtual resources from phys-
ical ones using resource allocation and scheduling mecha-
nisms, so that each virtual host receives its expected share of
the resource in question. Likewise, the virtual host abstrac-
tion limits the scope of logical resources to inside the box ,
so that each virtual host can safely manipulate its own log-
ical resources. The virtualization layer may not virtualize
all possible resources, and so there may be resources resid-
2