A Platform for High Performance and Flexible Virtual
Routers on Commodity Hardware
Norbert Egi, Micka-el Hoerdt, Laurent Mathy, Panagiotis Papadimitriou
Computing Dept., Lancaster University, UK
{n.egi, m.hoerdt, l.mathy, p.papadimitriou}@lancaster.ac.uk
Adam Greenhalgh, Mark Handley
Dept. of Computer Science, University College London, UK
{a.greenhalgh, m.handley}@cs.ucl.ac.uk
Felipe Huici
NEC Europe Ltd, Germany
felipe.huici@nw.neclab.eu
ABSTRACT
Multi-core CPUs, along with recent advances in memory and buses,
render commodity hardware a strong candidate for software router
virtualization. In this context, we present the design of a new plat-
form for virtual routers on modern PC hardware. We further discuss
our design choices in order to achieve both high performance and
flexibility for packet processing.
Categories and Subject Descriptors
C.2.6 [Computer Communication Networks]: [Internetworking–
Routers]
General Terms
Design
Keywords
Virtualization, Routers, Commodity Hardware
1. INTRODUCTION
Recent research has shown that modern PCs are a viable platform
for the implementation of high-performance software routers [1, 2].
The functionality provided by software routers combined with re-
cent virtualization technologies allows multiple instances of routers
to run concurrently on a single box while offering highly config-
urable forwarding planes and custom routing protocols. Some re-
cent PC-based virtual router prototypes [3] have been proposed,
but none of them exploit recent advances in commodity hardware
such as multi-core CPUs or network interface cards with hardware
multi-queuing.
In [1] we showed that the main performance bottleneck for PC-
based software routers is main memory access. With the memory
subsystem as the limiting factor for at least the next few years, the
number of spare CPU cycles is likely to increase. To exploit these
resources we can certainly consolidate a set of software routers run-
ning concurrently on the same hardware, inline with conventional
server virtualization. However, the short-lived nature of packets in-
side a router makes router virtualization challenging and certainly
more demanding. Is it possible to have the best of two worlds,
building a flexible virtual router platform that also reaches the per-
formance limits of the underlying hardware?
Motivated by this challenge, we present a new platform for soft-
ware virtual routers on commodity hardware. The platform lever-
ages modern and emerging hardware trends to provide: (i) con-
solidation of virtual data planes onto a common forwarding do-
main, (ii) highly configurable forwarding planes for advanced pro-
grammability, and (iii) the forwarding tree
1
as the basic resource al-
location unit for the mapping of virtual router components to cache
hierarchies.
2. PLATFORM DESIGN OVERVIEW
The design of our virtual router platform is mainly driven by
performance and flexibility. We use Xen’s [4] paravirtualization
to host the guest domains and Click [5] for packet processing and
forwarding.
The platform comprises the following basic components (Fig. 1):
(i) a management domain for the management of guest domains,
(ii) an isolated driver domain (IDD) for aggregated packet forward-
ing, and (iii) a number of guest domains for hosting control planes
(one per virtual router), and optionally forwarding planes when in-
creased isolation and safety properties are required.
The IDD is a virtual machine that has physical devices mapped
to it and runs the appropriate device driver and router software re-
quired to host our forwarding planes (FPs). The IDD hosts the
merged forwarding paths and provides the ability to control and
configure the individual FPs to their respective guest domains. The
merging process enables the consolidation of all FPs within the
IDD, allowing a large number of virtual routers to share common
network interfaces. In [1], we showed that forwarding within a
common domain provides significantly higher performance than
forwarding in the separate guest domains by avoiding costly per-
packet hypervisor domain context switches. Fig. 1 depicts this con-
figuration with FP1 running in the IDD.
The platform supports two additional packet forwarding config-
urations: (i) splitting a FP between an IDD and a separate guest
domain while using local I/O channels for inter-domain communi-
cation (e.g., FP2 and FP2’ in Fig. 1); and (ii) mapping interfaces
directly into guest domains so that each FP resides in a separate
guest domain (e.g., FP3 in Fig. 1). Configuration (i) can be used to
safely run FPs that include untrusted Click elements without com-
promising the performance and safety of other virtual routers’ FPs.
1
A forwarding tree is the set of packet processing elements neces-
sary to move a packet from a single input to all possible outputs.
ACM SIGCOMM Computer Communication Review 127 Volume 40, Number 1, January 2010