Kaleidoscope: Cloud Micro-Elasticity via VM State Coloring
Roy Bryant1 Alexey Tumanov1 Olga Irzak1 Adin Scannell1
Kaustubh Joshi2 Matti Hiltunen2 H. Andre´s Lagar-Cavilla2 Eyal de Lara1
1University of Toronto, 2AT&T Labs Research
Abstract
We introduce cloud micro-elasticity, a new model for cloud
Virtual Machine (VM) allocation and management. Current
cloud users over-provision long-lived VMs with large mem-
ory footprints to better absorb load spikes, and to conserve
performance-sensitive caches. Instead, we achieve elastic-
ity by swiftly cloning VMs into many transient, short-lived,
fractional workers to multiplex physical resources at a much
finer granularity. The memory of a micro-elastic clone is a
logical replica of the parent VM state, including caches, yet
its footprint is proportional to the workload, and often a frac-
tion of the nominal maximum. We enable micro-elasticity
through a novel technique dubbed VM state coloring, which
classifies VM memory into sets of semantically-related re-
gions, and optimizes the propagation, allocation and dedu-
plication of these regions. Using coloring, we build Kalei-
doscope and empirically demonstrate its ability to create
micro-elastic cloned servers. We model the impact of micro-
elasticity on a demand dataset from AT&T’s cloud, and show
that fine-grained multiplexing yields infrastructure reduc-
tions of 30% relative to state-of-the art techniques for man-
aging elastic clouds.
Categories and Subject Descriptors D.4.7 [Operating Sys-
tems]: Organization and Design – Distributed Systems;
D.4.1 [Operating Systems]: Process Management – Multi-
processing Multiprogramming Multitasking
General Terms Design, Experimentation, Measurement,
Performance
Keywords Virtualization, Cloud Computing
1. Introduction
Cloud computing caters to bursty Internet workloads with a
utility model that emphasizes pay-per-use and elasticity of
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. To copy otherwise, to republish, to post on servers or to redistribute
to lists, requires prior specific permission and/or a fee.
Eurosys ’11 April 10–13, Salzburg, Austria.
Copyright c© 2011 ACM 978-1-4503-0634-8/11/04. . . $10.00
provisioning. As with all utilities, there is a granularity as-
sociated with service delivery and billing. For Infrastructure
as a Service (IaaS) clouds, the granule is the virtual machine
(VM). Adopting virtualization as a building block yields dis-
tinct advantages in security, isolation and ease of manage-
ment, but this coarse granularity imposes inefficient patterns
that harm both users and providers.
In an ideal cloud, ‘elastic’ servers grow and shrink in
tight concert with user demand. Currently, a load balancer
adjusts the size of a pool of full-sized worker VMs [Amazon
a;b] that are booted from scratch from a template. Unfor-
tunately, this heavy-weight mechanism is a poor match for
the operation model of an efficient utility. Creation is slow –
new servers take a while to boot because it’s a laborious I/O
bound task. Moreover, this latency is hard to predict – instan-
tiation latencies in Amazon’s EC2 cloud have been observed
to fluctuate sharply around a two-minute mean [Hyperic].
Furthermore, once booted, the server’s performance-critical
application and OS caches are essentially empty, which de-
grades performance when it is most needed to service de-
mand spikes. Finally, VMs claim a full memory footprint
even if they are required for only short periods of time and
much of their memory is not actually used.
Therefore, server owners have incentives to keep VMs
active for long periods, both to provide slack resources dur-
ing long instantiation latencies, and because servers with
large, warm buffers become crucial to overall performance
and are too valuable to sacrifice. This practice may explain
why the proportion of EC2 ‘Extra Large’ instances (15 GiBs
of RAM) has grown from 12% to 56% in a year, even while
the total number of servers has tripled. Further, the propor-
tion of servers running longer than a month has nearly dou-
bled [RightScale]. These behaviors detract from the cloud
vision of matching resource usage to actual demand, inflate
user costs while still failing to achieve a good QoS if the
load exceeds expectations, and curtail providers’ ability to
consolidate and optimize infrastructure use.
This paper proposes a vision of cloud micro-elasticity,
in which cloud server elasticity is achieved through short-
lived, transient clone VMs, which are copies of a running
VM instance and allocate resources (memory, disk) only on
demand. To enable this, we introduce color-based fractional

VM cloning, a new technique that allows the fine-grained
management of VM state, and enables the swift instantiation
of stateful VMs that allocate resources in proportion to use.
By cloning a warm, running VM instead of booting a new
one, our workers inherit their parent VM’s state and do
not require warming. They come online faster, reach peak
performance sooner, and because short-lived worker VMs
typically access only a fraction of their state, they can service
transient spikes in load from within a smaller footprint.
Color-based fractional VM cloning uses a novel VM state
replication technique. Instead of blindly treating the VM as
a uniform collection of pages, it bridges the semantic gap
between the Virtual Machine Monitor (VMM) and the guest
OS by examining architectural information (e.g., page table
entries) and other clues to glean a more detailed understand-
ing of the guest’s state. This higher-quality knowledge al-
lows the VMM to optimize the propagation of state to clones
by identifying semantically related regions. Specifically, we
use VM state coloring to tailor the prefetching of kernel vs
user space regions, code vs data regions, and to optimize the
propagation of the file system page cache. Finally, coloring
provides hints that guide memory consolidation by identify-
ing regions with a high likelihood of content similarity.
To evaluate the performance of color-based fractional
VM cloning, we implemented Kaleidoscope, an elastic
server that reacts to transient load spikes by spawning frac-
tional VMs. Experiments using elastic Web and Online An-
alytical Processing (OLAP) workloads show that Kaleido-
scope significantly improves on the current state of the art.
First, Kaleidoscope instantiates new stateful clones in sec-
onds, and nearly matches the runtime performance of an ide-
alized cloning strategy that uses zero-latency eager full state
replication. Second, by bridging the semantic gap, Kaleido-
scope is effective in finding state that is likely to be needed
by the new clone. For example, for the OLAP workload,
it achieves 2.9 times the query throughput with 43% less
waste than color-blind cloning. Third, Kaleidoscope’s frac-
tional VM workers grow only as needed to satisfy new allo-
cations or hold newly transferred state. In our experiments,
the memory footprint of workers reached only 40% to 90%
of their parent’s memory allocation.
To further evaluate the advantages of color-based frac-
tional VM cloning, we simulated its effects on traces col-
lected from AT&T’s hosting operation. The simulation
shows that the finer-grained handling of VM state drasti-
cally reduces infrastructure use. With VM cloning to rapidly
create new workers, a server can scale faster, and therefore
requires much less slack capacity to deal with load increases.
Also, the reduced memory footprint translates into a denser
packing of VMs on physical infrastructure. The net result is
a 30% reduction in infrastructure, which creates energy and
money savings that can be shared with end-users via more
attractive fine-grained pricing schemes.
0
200
400
600
800
1000
1200
0 12 24 36 48 60
A
gg
re
ga
te
C
P
U
D
em
an
d
(p
er
ce
nt
)
Time (hours)
A B C
Figure 1. Aggregate CPU demand for sample customers
0.00%
20.00%
40.00%
60.00%
80.00%
100.00%
0
200
400
600
800
1000
1200
1400
1600
1800
2000
0 10 20 30 40 50 60
F
re
qu
en
cy

Clone Lifetime (min)
Frequency
Cumulative %
Figure 2. Elastic workers are typically short-lived
We also discuss the ways in which Kaleidoscope ensures
correct and consistent behavior. Although VM cloning is not
universally applicable without tuning, many legacy server
applications, including the OLAP database and Ecommerce
Apache Web server used in our evaluation, function cor-
rectly without modification.
This paper makes the following contributions. First, we
introduce the notion of VM state coloring as a general mech-
anism to bridge the semantic gap and glean high-quality in-
formation on the runtime state of a VM. Second, we show
how state coloring can be implemented efficiently by ex-
ploiting x86 architectural properties and guest kernel in-
trospection extensions. Third, we present Kaleidoscope, a
micro-elastic server that uses state coloring to optimize the
replication and sharing of VM state, and delivers a QoS that
approaches that of fully over-provisioned servers, while con-
suming resources proportional to the immediate demand.
And fourth, we quantify the benefits of deploying Kalei-
doscope servers using a data set of multi-customer demand
extracted from AT&T’s hosting operation, showcasing sub-
stantial savings for users and providers.
2. Real Data Motivates Micro-Elasticity
To use resources efficiently, elastic servers should grow and
shrink in tight concert with user demand. We examine the
potential for such elasticity using a month of demand data
from AT&T’s hosting operation (see Section 9 for more de-
tails about the data traces). AT&T hosting is a traditional
hosting business in which customers buy rack-space to stat-