Clairvoyant: A Comprehensive Source-Level Debugger for
Wireless Sensor Networks
Jing Yang, Mary Lou Soffa, Leo Selavo and Kamin Whitehouse
Department of Computer Science
University of Virginia
{jy8y, soffa, selavo, whitehouse}@cs.virginia.edu
Abstract
Wireless sensor network (WSN) applications are notori-
ously difficult to develop and debug. This paper describes
Clairvoyant which is a comprehensive source-level debugger
for wireless, embedded networks. With Clairvoyant, a devel-
oper can wirelessly connect to a sensor network and execute
standard debugging commands including break, step, watch,
and backtrace, as well as new commands that are specially
designed for debugging WSNs. Clairvoyant attempts to min-
imize its effect on the program being debugged in terms of
network load, memory footprint, execution speed, clock con-
sistency, and flash lifetime.
Categories and Subject Descriptors
D.2.5 [Software Engineering]: Testing and Debug-
ging—debugging aids, distributed debugging
General Terms
Design, Experimentation, Performance
Keywords
Source-Level Debugging, Distributed Debugging, Em-
bedded Debugging, Wireless Sensor Networks
1 Introduction
Source-level debuggers allow a developer to execute a
program one statement or code block at a time and to watch
the program as it is being executed. They are called source-
level debuggers because they operate on symbols and state-
ments defined in the program’s source code. For example,
the developer can set a breakpoint on a line of code, observe
or modify a live variable, and invoke an arbitrary function
using the symbol names defined in the source code. Also
known as symbolic debuggers,oroftensimplydebuggers,
these tools are an essential part of most modern debugging
environments. They make software development much eas-
ier, to the point that a good debugger can sway a program-
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. To copy otherwise, to republish, to post on servers or to redistribute
to lists, requires prior specific permission and/or a fee.
SenSys’07, November 6–9, 2007, Sydney, Australia.
Copyright 2007 ACM 1-59593-763-6/07/0011 ...$5.00
mer’s decision to use a particular language [26] or hardware
platform [29].
Remote debugging is the process in which a host machine
is used to debug a process on a target machine. Remote de-
buggers have existed for many years and are commonly used
to debug single embedded devices but, as we discuss in Sec-
tion 2, cannot be easily applied to wireless sensor networks
(WSNs). This is because hardware debuggers are expensive
and sometimes impossible to deploy in large numbers or over
large geographic areas, while software approaches do not ac-
comodate the multi-hop, resource-constrained, and timing-
dependent nature of WSNs. The most common approach to
debug a WSN today is to use a simulator for large-scale net-
works and a hardware debugger for low-level device drivers.
This is problematic because many bugs only manifest them-
selves when deployed on real hardware, in a real and noisy
environment, and at large scale. The lack of a source-level
debugger for real WSN deployments increases development
time by making it difficult to diagnosis and fix these bugs. It
also makes it difficult to verify correct operation, increasing
the chance that buggy code will survive the testing process
and make its way into production code.
In this paper, we present a system called Clairvoyant that
provides comprehensive source-level debugging solution for
WSNs. It can operate on the deployment hardware and in the
deployment environment, without requiring any additional
hardware or wires. Furthermore, no modifications need to
be made to the application’s source code. Clairvoyant pro-
vides a suite of standard debugging commands like break,
step, watch,andbacktrace, among others. It also provides
new functionality that is specially designed for WSNs, in-
cluding access to typical hardware such as external flash and
interrupt vectors, global commands such as gstop and gcon-
tinue that operate not on a single node but on all nodes in the
network, and the ability to log variables to LEDs, RAM, ex-
ternal flash, or radio. These commands are described in more
detail in Section 3. Clairvoyant can be used to debug race
conditions, stack overflow, errors in global state, and errors
in low-level hardware drivers, all of which are common yet
difficult-to-debug errors in WSNs. It has been implemented
on the Mica2 sensor node and is demonstrated in Section 6
through two case studies on real, documented bugs from the
TinyOS repository: stack overflow and deadlock.
The process of debugging will always affect how a pro-
gram executes, and Clairvoyant attempts to minimize this
189

Realism
S
c
a
l
e
Deployment
ICE
Simulation
EmStar
Figure 1. Existing source-level debugging tools for WSNs
cannot deal with scale and reality simultaneously.
effect in order to reduce the number of so-called Heisenbugs
– those bugs that change or disappear once a debugger is
used. To this end, Clairvoyant implements debugging com-
mands by modifying the target binary, an approach called
dynamic binary instrumentation. This is the only approach
that runs the target program at native speed directly on the
MCU, or “on the metal”, without using extra hardware or
making changes to the program’s source code. However,
sensor nodes have limited space for debugging software and
have flash-based program memory, which limits the number
of modifications that can be made to the target binary. In Sec-
tion 4, we describe the techniques we developed to make dy-
namic binary instrumentation work on resource-constrained
sensor nodes. In Section 5, we explore and empirically eval-
uate the costs, trade-offs and limitations of these techniques
in terms of network load, memory footprint, execution speed,
clock consistency, and flash lifetime.
2 Background and Related Work
Debugging is a multi-step process that requires several
tools to collaborate with each other, including source-level
debuggers. For example, tools like Tarantula help with fault
localization by identifying the lines of code most highly cor-
related with failed test cases [9]. Integrated development en-
vironments (IDEs) like Eclipse [18] and Visual Studio [13]
help with fix determination by suggesting possible changes
to syntactical errors in the code. Dynamic reprogramming
systems like Deluge help with a process called fix applica-
tion [8]. The main goal of source-level debuggers is called
cause identification, in which the developer tests a hypothe-
sis about the cause of a bug.
Source-level debuggers such as GDB [21] and DBX [20]
are typically implemented in one of four ways. Most de-
buggers use the processor’s built-in hardware support: spe-
cial registers store the locations of breakpoints or variable
watches and, when the program counter or a memory ac-
cess matches these registers, the processor halts execution
and transfers control to the debugger. This approach cannot
be used for WSNs because the 8-bit microcontroller units
(MCUs) most commonly used on sensor nodes do not pro-
vide hardware support for software debuggers. A second ap-
proach is to control the run-time system through APIs like
the Java Debugger API [19] and the ICorDebug API [14]
for the .NET Common Language Runtime. However, most
WSN applications do not run in this type of run-time sys-
tem. Smaller virtual machines, software dynamic translators,
or interpreters [7] could be used to implement a debugger
Figure 2. The Clairvoyant user interface consists of mul-
tiple GDB-like terminals.
for WSNs, but we did not employ these techniques because
they can significantly change the temporal characteristics of
a program and therefore increase the number of heisenbugs.
A third approach is hardware emulation through an in-circuit
emulator (ICE) such as the AVR JTAG ICE [2], which is
an external piece of hardware that interfaces with GDB and
controls firmware execution by physically attaching to the
I/O pins on the MCU. ICEs are most commonly used for de-
bugging embedded devices, but they require a physical wire
and additional hardware for every node and thus do not scale
with the number of nodes or the geographic size of a net-
work. Furthermore, they cannot be used spontaneously be-
cause the hardware must be deployed in advance. A fourth
approach is to implement debugging commands in software,
through modification of the target executable. For example,
a breakpoint can be implemented by inserting a trap into the
binary executable; whenever the trap is reached, control will
be transferred to the debugger. This approach may be used
when the developer creates more breakpoints or watches a
larger section of memory than the hardware supports, and is
most similar to the approach used by Clairvoyant. To the
best of our knowledge, Clairvoyant is the first implementa-
tion of this approach on a platform with flash-based instead
of RAM-based program memory.
The most common way to employ the source-level de-
bugging technology described above in a WSN application
is through simulation, which allows the application to be ex-
ecuted and debugged on a machine with full OS and hard-
ware debugging support [10, 16, 24]. Simulation makes it
easy to watch program execution, and can be used to de-
bug a large network with 1000s of nodes. However, all sen-
sor readings and radio communication are based on models
that can never capture the full complexity of the real world.
This problem is ameliorated by a system called EmStar [6],
which runs the application in a simulation-like environment,
but uses real-world I/O by connecting the simulation to the
sensors and radios of a real-deployed sensor network. Em-
Star requires a physical wire to be run to each node, which
limits the maximum size and geographic area of any network
to be debugged. Furthermore, EmStar affects the speed with
which the application executes by running the application on
a high-powered, multi-tasking processor and by introducing
190