Palacios and Kitten: New High Performance Operating Systems For
Scalable Virtualized and Native Supercomputing
John Lange∗, Kevin Pedretti†, Trammell Hudson†, Peter Dinda∗, Zheng Cui‡, Lei Xia∗, Patrick Bridges‡,
Andy Gocke∗, Steven Jaconette∗, Mike Levenhagen†, and Ron Brightwell†
∗ Northwestern University, Department of Electrical Engineering and Computer Science
Email: {jarusl,pdinda,leixia,agocke,jaconette}@northwestern.edu
† Sandia National Laboratories, Scalable System Software Department
Email: {ktpedre,mjleven,rbbrigh}@sandia.gov, hudson@osresarch.net
‡ University of New Mexico, Department of Computer Science
Email: {zheng,bridges}@cs.unm.edu
Abstract—Palacios is a new open-source VMM under de-
velopment at Northwestern University and the University of
New Mexico that enables applications executing in a virtualized
environment to achieve scalable high performance on large ma-
chines. Palacios functions as a modularized extension to Kitten,
a high performance operating system being developed at Sandia
National Laboratories to support large-scale supercomputing
applications. Together, Palacios and Kitten provide a thin layer
over the hardware to support full-featured virtualized environ-
ments alongside Kitten’s lightweight native environment. Pala-
cios supports existing, unmodified applications and operating
systems by using the hardware virtualization technologies in
recent AMD and Intel processors. Additionally, Palacios lever-
ages Kitten’s simple memory management scheme to enable
low-overhead pass-through of native devices to a virtualized
environment. We describe the design, implementation, and
integration of Palacios and Kitten. Our benchmarks show that
Palacios provides near native (within 5%), scalable perfor-
mance for virtualized environments running important parallel
applications. This new architecture provides an incremental
path for applications to use supercomputers, running special-
ized lightweight host operating systems, that is not significantly
performance-compromised.
Keywords-virtual machine monitors; lightweight kernels;
parallel computing; high performance computing
I. INTRODUCTION
This paper introduces Palacios, a new high performance
virtual machine monitor (VMM) architecture, that has been
embedded into Kitten, a high performance supercomputing
operating system (OS). Together, Palacios and Kitten pro-
vide a flexible, high performance virtualized system software
platform for HPC systems. This platform broadens the
applicability and usability of HPC systems by:
This project is made possible by support from the National Science Foun-
dation (NSF) via grants CNS-0709168, CNS-0707365, and the Department
of Energy (DOE) via a subcontract from Oak Ridge National Laboratory
on grant DE-AC05-00OR22725. John Lange was partially supported by a
Symantec Research Labs Fellowship. Sandia is a multiprogram laboratory
operated by Sandia Corporation, a Lockheed Martin Company, for the
United States Department of Energy’s National Nuclear Security Adminis-
tration under contract DE-AC04-94AL85000.
• providing access to advanced virtualization features
such as migration, full system checkpointing, and de-
bugging;
• allowing system owners to support a wider range of
applications and to more easily support legacy appli-
cations and programming models when changing the
underlying hardware platform;
• enabling system users to incrementally port their codes
from small-scale development systems to large-scale
supercomputer systems while carefully balancing their
performance and system software service requirements
with application porting effort; and
• providing system hardware and software architects with
a platform for exploring hardware and system software
enhancements without disrupting other applications.
Palacios is a “type-I” pure VMM [1] under development at
Northwestern University and the University of New Mexico
that provides the ability to virtualize existing, unmodified
applications and their operating systems with no porting.
Palacios is designed to be embeddable into other operating
systems, and has been embedded in two so far, including Kit-
ten. Palacios makes extensive, non-optional use of hardware
virtualization technologies and thus can scale with improved
implementations of those technologies.
Kitten is an OS being developed at Sandia National
Laboratories that is being used to investigate system software
techniques for better leveraging multicore processors and
hardware virtualization in the context of capability super-
computers. Kitten is designed in the spirit of lightweight
kernels [2], such as Sandia’s Catamount [3] and IBM’s
CNK [4], that are well known to perform better than
commodity kernels for HPC. The simple framework pro-
vided by Kitten and other lightweight kernels facilitates
experimentation, has led to novel techniques for reducing
the memory bandwidth requirements of intra-node message
passing [5], and is being used to explore system-level options
for improving resiliency to hardware faults.