Live Migration of Virtual Machines Christopher Clark, Keir Fraser, Steven Hand, Jacob Gorm Hansen���, Eric Jul���, Christian Limpach, Ian Pratt, Andrew Warfield University of Cambridge Computer Laboratory ��� Department of Computer Science 15 JJ Thomson Avenue, Cambridge, UK University of Copenhagen, Denmark firstname.lastname@cl.cam.ac.uk {jacobg,eric}@diku.dk Abstract Migrating operating system instances across distinct phys- ical hosts is a useful tool for administrators of data centers and clusters: It allows a clean separation between hard- ware and software, and facilitates fault management, load balancing, and low-level system maintenance. By carrying out the majority of migration while OSes con- tinue to run, we achieve impressive performance with min- imal service downtimes we demonstrate the migration of entire OS instances on a commodity cluster, recording ser- vice downtimes as low as 60ms. We show that that our performance is sufficient to make live migration a practical tool even for servers running interactive loads. In this paper we consider the design options for migrat- ing OSes running services with liveness constraints, fo- cusing on data center and cluster environments. We intro- duce and analyze the concept of writable working set, and present the design, implementation and evaluation of high- performance OS migration built on top of the Xen VMM. 1 Introduction Operating system virtualization has attracted considerable interest in recent years, particularly from the data center and cluster computing communities. It has previously been shown [1] that paravirtualization allows many OS instances to run concurrently on a single physical machine with high performance, providing better use of physical resources and isolating individual OS instances. In this paper we explore a further benefit allowed by vir- tualization: that of live OS migration. Migrating an en- tire OS and all of its applications as one unit allows us to avoid many of the difficulties faced by process-level mi- gration approaches. In particular the narrow interface be- tween a virtualized OS and the virtual machine monitor (VMM) makes it easy avoid the problem of ���residual de- pendencies��� [2] in which the original host machine must remain available and network-accessible in order to service certain system calls or even memory accesses on behalf of migrated processes. With virtual machine migration, on the other hand, the original host may be decommissioned once migration has completed. This is particularly valuable when migration is occurring in order to allow maintenance of the original host. Secondly, migrating at the level of an entire virtual ma- chine means that in-memory state can be transferred in a consistent and (as will be shown) efficient fashion. This ap- plies to kernel-internal state (e.g. the TCP control block for a currently active connection) as well as application-level state, even when this is shared between multiple cooperat- ing processes. In practical terms, for example, this means that we can migrate an on-line game server or streaming media server without requiring clients to reconnect: some- thing not possible with approaches which use application- level restart and layer 7 redirection. Thirdly, live migration of virtual machines allows a sepa- ration of concerns between the users and operator of a data center or cluster. Users have ���carte blanche��� regarding the software and services they run within their virtual machine, and need not provide the operator with any OS-level access at all (e.g. a root login to quiesce processes or I/O prior to migration). Similarly the operator need not be concerned with the details of what is occurring within the virtual ma- chine instead they can simply migrate the entire operating system and its attendant processes as a single unit. Overall, live OS migration is a extremelely powerful tool for cluster administrators, allowing separation of hardware and software considerations, and consolidating clustered hardware into a single coherent management domain. If a physical machine needs to be removed from service an administrator may migrate OS instances including the ap- plications that they are running to alternative machine(s), freeing the original machine for maintenance. Similarly, OS instances may be rearranged across machines in a clus- ter to relieve load on congested hosts. In these situations the combination of virtualization and migration significantly improves manageability.

We have implemented high-performance migration sup- port for Xen [1], a freely available open source VMM for commodity hardware. Our design and implementation ad- dresses the issues and tradeoffs involved in live local-area migration. Firstly, as we are targeting the migration of ac- tive OSes hosting live services, it is critically important to minimize the downtime during which services are entirely unavailable. Secondly, we must consider the total migra- tion time, during which state on both machines is synchro- nized and which hence may affect reliability. Furthermore we must ensure that migration does not unnecessarily dis- rupt active services through resource contention (e.g., CPU, network bandwidth) with the migrating OS. Our implementation addresses all of these concerns, allow- ing for example an OS running the SPECweb benchmark to migrate across two physical hosts with only 210ms un- availability, or an OS running a Quake 3 server to migrate with just 60ms downtime. Unlike application-level restart, we can maintain network connections and application state during this process, hence providing effectively seamless migration from a user���s point of view. We achieve this by using a pre-copy approach in which pages of memory are iteratively copied from the source machine to the destination host, all without ever stopping the execution of the virtual machine being migrated. Page- level protection hardware is used to ensure a consistent snapshot is transferred, and a rate-adaptive algorithm is used to control the impact of migration traffic on running services. The final phase pauses the virtual machine, copies any remaining pages to the destination, and resumes exe- cution there. We eschew a ���pull��� approach which faults in missing pages across the network since this adds a residual dependency of arbitrarily long duration, as well as provid- ing in general rather poor performance. Our current implementation does not address migration across the wide area, nor does it include support for migrat- ing local block devices, since neither of these are required for our target problem space. However we discuss ways in which such support can be provided in Section 7. 2 Related Work The Collective project [3] has previously explored VM mi- gration as a tool to provide mobility to users who work on different physical hosts at different times, citing as an ex- ample the transfer of an OS instance to a home computer while a user drives home from work. Their work aims to optimize for slow (e.g., ADSL) links and longer time spans, and so stops OS execution for the duration of the transfer, with a set of enhancements to reduce the transmitted image size. In contrast, our efforts are concerned with the migra- tion of live, in-service OS instances on fast neworks with only tens of milliseconds of downtime. Other projects that have explored migration over longer time spans by stop- ping and then transferring include Internet Suspend/Re- sume [4] and ��Denali [5]. Zap [6] uses partial OS virtualization to allow the migration of process domains (pods), essentially process groups, us- ing a modified Linux kernel. Their approach is to isolate all process-to-kernel interfaces, such as file handles and sock- ets, into a contained namespace that can be migrated. Their approach is considerably faster than results in the Collec- tive work, largely due to the smaller units of migration. However, migration in their system is still on the order of seconds at best, and does not allow live migration pods are entirely suspended, copied, and then resumed. Further- more, they do not address the problem of maintaining open connections for existing services. The live migration system presented here has considerable shared heritage with the previous work on NomadBIOS [7], a virtualization and migration system built on top of the L4 microkernel [8]. NomadBIOS uses pre-copy migration to achieve very short best-case migration downtimes, but makes no attempt at adapting to the writable working set behavior of the migrating OS. VMware has recently added OS migration support, dubbed VMotion, to their VirtualCenter management software. As this is commercial software and strictly disallows the publi- cation of third-party benchmarks, we are only able to infer its behavior through VMware���s own publications. These limitations make a thorough technical comparison impos- sible. However, based on the VirtualCenter User���s Man- ual [9], we believe their approach is generally similar to ours and would expect it to perform to a similar standard. Process migration, a hot topic in systems research during the 1980s [10, 11, 12, 13, 14], has seen very little use for real-world applications. Milojicic et al [2] give a thorough survey of possible reasons for this, including the problem of the residual dependencies that a migrated process re- tains on the machine from which it migrated. Examples of residual dependencies include open file descriptors, shared memory segments, and other local resources. These are un- desirable because the original machine must remain avail- able, and because they usually negatively impact the per- formance of migrated processes. For example Sprite [15] processes executing on foreign nodes require some system calls to be forwarded to the home node for execution, leading to at best reduced perfor- mance and at worst widespread failure if the home node is unavailable. Although various efforts were made to ame- liorate performance issues, the underlying reliance on the availability of the home node could not be avoided. A sim- ilar fragility occurs with MOSIX [14] where a deputy pro- cess on the home node must remain available to support remote execution.