Towards High Performance Virtual Routers
on Commodity Hardware
Norbert Egi
Lancaster University, UK
n.egi@lancaster.ac.uk
Adam Greenhalgh
University College London, UK
a.greenhalgh@cs.ucl.ac.uk
Mark Handley
University College London, UK
m.handley@cs.ucl.ac.uk
Mickael Hoerdt
Lancaster University, UK
m.hoerdt@lancaster.ac.uk
Felipe Huici
NEC Europe Ltd, Germany
felipe.huici@nw.neclab.eu
Laurent Mathy
Lancaster University, UK
l.mathy@lancaster.ac.uk
ABSTRACT
Modern commodity hardware architectures, with their mul-
tiple multi-core CPUs and high-speed system interconnects,
exhibit tremendous power. In this paper, we study perfor-
mance limitations when building both software routers and
software virtual routers on such systems. We show that the
fundamental performance bottleneck is currently the mem-
ory system, and that through careful mapping of tasks to
CPU cores, we can achieve forwarding rates of 7 million
minimum-sized packets per second on mid-range server-class
systems, thus demonstrating the viability of software routers.
We also find that current virtualisation systems, when used
to provide forwarding engine virtualisation, yield aggregate
performance equivalent to that of a single software router, a
tenfold improvement on current virtual router platform per-
formance. Finally, we identify principles for the construc-
tion of high-performance software router systems on com-
modity hardware, including full router virtualisation sup-
port.
1. INTRODUCTION
Over the last few years virtualisation has become a hot
topic, with platforms such as Xen[1] and VMware[2] en-
abling virtual machines on regular x86 PC hardware, and
Intel and AMD both adding virtualisation extensions[3] to
their processors. Of course, virtualisation is nothing new:
IBM’s CP/CMS[4] provided virtual machine support in the
late 1960s. However, only recently has PC hardware be-
come powerful enough to make running multiple virtual ma-
chines on one inexpensive box a practical proposition. From
a server point of view, virtualisation makes a great deal of
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, to
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee.
ACM CoNEXT 2008, December 10-12, 2008, Madrid, SPAIN
Copyright 2008 ACM 978-1-60558-210-8/08/0012 ...$5.00.
sense: a single machine in a data center can support many
different network servers. One of the advantages arising
from this is isolation, ensuring that if a virtual server is com-
promised the damage is limited and the faulty server does not
exhaust all OS resources. Another clear advantage is that un-
used resources from one server can be used by another. And
perhaps most importantly, different administrators can man-
age different servers on the same hardware without needing
to trust each other, thus enabling many new business models.
The advantages of isolation and independent administra-
tion carry over to network virtualisation. Virtual LANs
(VLANs) and Virtual Private Networks (VPNs) allow a sin-
gle network to be subdivided and to have different users of
the network isolated from each other. However, while most
ethernet switches support VLANs, the model is that a single
switch administrator configures these VLANs. While from
the outside it looks like the switch is behaving as a separate
switch for each VLAN, the switch itself is not virtualized in
any real sense. The same is true with VPNs: an ISP might
sell a VPN service, allowing a customer to interconnect his
sites over the ISP’s Internet backbone, safe in the knowledge
that they are isolated from other users of the public Internet.
However, the customer does not get a virtual slice of the
ISP’s core routers to manage as he sees fit.
Extending the idea of true virtualisation to network re-
sources, and to routers in particular, seems like the natural
next step. The benefits are obvious: a single virtual router
platform can provide independent routing for multiple net-
works in a manner that permits independent management of
those networks. There are many applications for such a tech-
nology. For example, within our university there is a router
connected to the campus backbone that also provides rout-
ing between IP subnets within our department. Should this
be managed by the campus network administrators or by our
department? Such petty turf wars are remarkably common-
place. Virtual routers allow separate administration within a
single box in a natural manner; they also enable many busi-
ness models that are currently difficult.
In the research world, the VINI [5] platform advocates