Xenprobes, A Lightweight User-space Probing Framework
for Xen Virtual Machine
Nguyen Anh Quynh, Kuniyasu Suzaki
National Institute of Advanced Industrial Science and Technology, Japan.
{nguyen.anhquynh,k.suzaki}@aist.go.jp
Abstract
This paper presents Xenprobes, a lightweight framework
to probe the guest kernels of Xen Virtual Machine. Xen-
probes is useful for various purposes such as as moni-
toring real-time status of production systems, analyzing
performance bottlenecks, logging specific events or trac-
ing problems of Xen-based guest kernel. Compared to
other kernel probe solutions, Xenprobes introduces some
unique advantages. To name a few: First, our framework
puts the the breakpoint handlers in user-space, so it is
significantly easier to develop and debug. Second, Xen-
probes allows to probe multiple guests at the same time.
Last but not least, Xenprobes supports all kind of Oper-
ating Systems supported by Xen.
1 Introduction
Testing and debugging Operating System (OS) kernel is a
hard and tired job of kernel developers. An easy and intu-
itive solution like inserting debug code (such as printk()
function in Linux) into the kernel source, then recompile
it and reboot the system is widely used. Nevertheless,
this technique has a major drawbacks: It is very time-
consuming and slow, especially because compiling ker-
nel might take no less than 30 minutes on old machines.
Moreover, in general commercial OS-es do not provide
the source code for us to modify and recompile in the
first place.
One solution to the problem is to use kernel debug-
ger [16] [10]. However, while kernel debuggers allow
developers to inspect kernel at run-time, a debugger is
not always desirable because it requires user interactiv-
ity. In case the testing and debugging must be done auto-
matically, for example to monitor system status in a long
time, this approach is not up to the task.
Hence the advent of dynamical probing technique,
which allows the developers - at run-time - to specify the
actions when corresponding events happen in the kernel.
Basically this technology dynanmically inserts a break-
point into a running kernel, without having to modify
the kernel source. The place of the breakpoint, usually
specified by its address in kernel address space, is called
probe-point. Each probe associates a probe-point with a
probe handler. A probe handler runs as extension to the
system breakpoint interrupt handler and usually has little
or no dependence on system facilities. Probes are able to
be inserted in the almost everywhere in the kernel, like
interrupt-time, task-time, disabled, inter-context switch
and SMP-enabled code paths, etc...
In Linux world, such a probe framework is Kprobes.
Kprobes was merged into Linux kernel from version
2.6.9, and provides a lightweight interface for kernel
modules to inject probes and register corresponding
probe handlers. Kprobes are intended to be used in test
and development environments. During test, faults may
be injected or simulated by the probing module. In de-
velopment, debugging code, like a printk() function, may
be easily inserted without having to recompile the kernel.
However, solutions like Kprobes have some major
shortcomings. Here are the most notable drawbacks:
1. Probe handlers come in the shape of kernel code,
specifically in kernel module. The problem is that
comparing to programming in user-space, program-
ming in kernel context is limited by many restric-
tions, such as allocating and accessing resource,
short on available library, no floating-point math,
etc... Programming for OS kernel is considered very
complicated and tricky, thus usually requires highly
experienced developers, because a broken code can
make the whole system in-stable.
That is a reason why programming in user-space
is always encouraged over kernel-space. In fact,
there is a basic principle within kernel developer
community: Whatever can be done in user-space,
never do it in kernel-space unless absolutely neces-
sarily. Various attempts from different open source
2007 USENIX Annual Technical ConferenceUSENIX Association 15

projects trying to submit their work to Linux kernel
are rejected, and asked to re-architecture their code
to work in user-space instead.
2. Kprobes cannot put the probes in some places in
the kernel, such as in the code that implements
Kprobes and functions like do page fault() and no-
tifier call chain() [8].
3. Kprobes makes no attempt to prevent probe han-
dlers from stepping on each other – for example
it is not a good idea to probe printk() and then
call printk() from inside the probe handler. If a
probe handler hits a probe, then the second probe’s
handlers will not run in that instance, and the
kprobe.nmissed member of the second probe will be
incremented [8].
4. Kprobes is designed and works for Linux only, thus
the probing code made for Kprobes cannot be easily
reused for other OS-es.
Recently virtual machine (VM) technology has
emerged as one of the hottest topics in computer re-
search. The principle of VM technology is to allow the
creation of many virtual hosts running at the same time
on the same physical machine, each running an instance
of an OS. Obviously VM software such as Xen Virtual
Machine [5] [14] can help to reduce both hardware and
maintenance costs for organizations that need to use vari-
ous machines for different services. Especially Xen even
offers a convenient method of debugging OS kernel, so
it is possible to debug an OS like debugging an user-
process [1].
Taking the advantage of Xen technology, this paper
proposes a framework named Xenprobes for probing
Xen-based guest OS kernel. Similar to Kprobes, Xen-
probes allows developers to dynamically inject probes
into guest VMs at run-time. However, Xenprobes is able
to address the above-mentioned problems of Kprobes:
1. Xenprobes’ handlers completely work in user-
space, therefore significantly easier than Kprobes
to develop the handlers, even with less experienced
programmers. This includes the benefit of using any
library available in user-space.
2. Xenprobes can put the probes at any place in probed
VMs without worrying about conflict.
3. Because the Xenprobes handlers run in user-space
instead of in the kernel of the probed VM, we elim-
inate the problem of stepping on other handlers like
in Kprobes’ case.
4. Xenprobes is OS-independent, and can provide ser-
vice for any OS supported by Xen. In addition, Xen-
probes is designed to support probing multiple VMs
at the same time.
The rest of this paper is organized as followings.
Section 2 briefly covers some background of Xen Vir-
tual Machine and Xen debugging technique. Section
3 presents the architecture and implementation of Xen-
probes framework, while section 4 discusses some is-
sues that can be raised when using the framework. Sec-
tion 5 evaluates the performance overhead of our frame-
work in Linux guest kernels. Section 6 summaries the
related works, and compare their advantage as well as
drawbacks with our approach. Finally we conclude the
paper in section 7.
2 Background on Xen Virtual Machine
Our framework Xenprobes is based on Xen, and ex-
ploits the debugging architecture of Xen to inject soft-
ware breakpoints into probed VM. In this part, we will
take a brief look at Xen technology. After that we dis-
cuss the kernel debugging architecture for Xen VM ver-
sion 3.0.3, the latest version as of this writing, which is
used by our Xenprobes.
2.1 Xen Virtual Machine
Xen is an open source virtual machine monitor initially
developed by the University of Cambridge Computer
Laboratory and now promoted by various industrial play-
ers like Intel, AMD, IBM, HP, RedHat, Novel and by the
open source community. Xen can be used to partition
a machine to support the concurrent execution of multi-
ple operating systems (OS). Xen is outstanding because
the performance overhead introduced by virtualization is
negligible: the slowdown is around 3% only ([4]). Var-
ious practices take the advantages offered by Xen, such
as server consolidation, co-located hosting facilities, dis-
tributed services and application mobility, as well as test-
ing and debugging software.
Basically, Xen is a thin layer of software running on
top the bare hardware. This layer is either called hy-
pervisor or virtual machine monitor. Its main job is to
provide a virtual machine abstraction to the above OS-
es. Running on top of Xen, VM is called Xen domain,
or domain in short. A privileged special domain named
Domain0 (or Dom0 in short) always runs. Dom0 controls
other domains (called User Domain, or DomU in short),
including jobs like start, shutdown, reboot, save, restore
and migrate them between physical machines. Espe-
cially, Dom0 is able to map and access to memory of
other DomUs at run-time.
2
2007 USENIX Annual Technical Conference USENIX Association16