Auton Robot (2009) 27: 313–325
DOI 10.1007/s10514-009-9137-8
An insightful comparison between experiments
in mobile robotics and in science
Francesco Amigoni · Monica Reggiani ·
Viola Schiaffonati
Received: 10 February 2009 / Accepted: 5 August 2009 / Published online: 19 August 2009
' Springer Science+Business Media, LLC 2009
Abstract Experiments are essential ingredients of science,
both to confirm/refute a theory and to discover new theories.
It is a common belief that experimentation in mobile robot-
ics has not yet reached a level of maturity comparable with
that reached in science, for example in physics, considered
as the paradigm of a mature, stable, and well-founded scien-
tific discipline. In this paper, starting from a representative
sample of the current state of the art, we identify some ba-
sic issues of experiments in mobile robot localization and
mapping. These issues, when viewed in the context of some
general principles about experiments in science and engi-
neering, lead us to derive some insightful considerations on
the role of experiments in mobile robotics. Reflecting the
background of the authors, the paper has an interdisciplinary
nature at the meeting point of mobile robotics and philoso-
phy of science.
Keywords Experiments · Localization · Mapping
1 Introduction
Starting from the Scientific Revolution of the XVII cen-
tury, natural scientists have developed the modern scien-
tific method, progressing from simple collections of obser-
vations to the extensive use of experiments for investigat-
ing natural phenomena. Experiments are so fundamental in
F. Amigoni () · V. Schiaffonati
Dipartimento di Elettronica e Informazione, Politecnico di
Milano, Milano, Italy
e-mail: francesco.amigoni@polimi.it
M. Reggiani
Dipartimento di Tecnica e Gestione dei Sistemi Industriali,
Università di Padova, Vicenza, Italy
science that the ‘scientific method’ coincides with the ‘ex-
perimental method’, at least in the natural sciences. Exper-
iments are performed for discovering new hypotheses that
explain natural phenomena, for confirming/refuting theo-
ries, and for choosing among competing hypotheses. The
scientific method, which has proved very successful for nat-
ural sciences, could also be useful in engineering, for exam-
ple when the behavior of a system or its performance are dif-
ficult to characterize analytically. Therefore, as we argue in
this paper, engineers have the valuable opportunity to learn
from the natural sciences, whose experimental methodolo-
gies have been developed and optimized along the centuries.
At the same time, they should also consider some fundamen-
tal differences in experimentation between natural sciences
and engineering.
It is a common belief that experimentation in mobile ro-
botics has not yet reached a level of maturity comparable
with that of experimentation in science and in other engi-
neering disciplines. Recently, the interest in experimental
methodologies increased dramatically within the mobile ro-
botics community, both from researchers and from funding
agencies, according to the idea that good experimental ac-
tivities could reduce the gap between research and industrial
applications. Some projects have been funded by the Eu-
ropean Community, including Rawseeds (2006) and RoSta
(2007), and series of workshops on benchmarks (EURON
GEM Sig 2007) and on measures for intelligent and au-
tonomous systems (Permis 2000) have been held in the latest
years. Also some papers on these topics started to appear in
major conferences (Amigoni et al. 2007).
Despite the ongoing efforts and the recognized impor-
tance of experiments for rigorously evaluating new ap-
proaches and for reporting them in an objective and com-
plete manner, these ideas have not yet become really part of

314 Auton Robot (2009) 27: 313–325
current practice, as it can be seen by having a look at re-
cently published papers. In our opinion, this is due, among
other reasons, to the difficulty to perform time-consuming
experiments under the publish-or-perish pressure and to the
weak awareness of experiments as fundamental elements in
the development of a robotic system.
In this paper, we argue in favor of this last issue, in the
context of mobile robot localization and mapping. Localiza-
tion is the process of estimating the pose of a robot in a map,
while mapping is the process of building a map of an envi-
ronment, by integrating perceptions taken at different poses.
These two processes are tightly correlated and are globally
referred to as SLAM (Simultaneous Localization And Map-
ping)(Thrunetal.2005). In the context of this special issue,
localization and mapping enable stable navigation solutions,
namely the ability of a robotic system to sense and create in-
ternal representations of its environment and estimate pose
(where pose consists of position and orientation) with re-
spect to a fixed coordinate frame. This provides mobile ro-
bots with the ability to identify obstacles and hazards in the
environment, and to maintain an estimate of where they are
and where they have been.
In this paper, we start by describing some experimental
trends that emerged from a survey of a significant sample of
papers in mobile robot localization and mapping (Sect. 2).
Then, we do an excursus over the very idea of experiment, as
it has been shaped in science and engineering, with the goal
of identifying some principles that define and characterize
experiments in general (Sect. 3). Finally, with these princi-
ples in mind, we go back to mobile robotics and we discuss
some insightful issues on the role of experiments in local-
ization and mapping (Sect. 4). Reflecting the background of
the authors, the paper has an interdisciplinary nature at the
intersection between mobile robotics and philosophy of sci-
ence.
The general purpose of this paper is to contribute to the
discussion on the definition of an experimental methodol-
ogy for mobile robotics. In this sense, our scope is broader
than the definition of benchmarks (that are anyway basic in-
gredients of a sound methodology, see Bonsignorio et al.
2007) and includes also a discussion on the purposes of ex-
perimental activities. However, we explicitly note that we
do not aim at proposing any definite experimental method-
ology, but only at giving a contribution in the direction of
this definition.
2 Trends in experimental activities in mobile robot
localization and mapping
In this section, we outline some interesting trends emerging
from experimental activities reported in published papers in
the area of mobile robot localization and mapping. These
trends will be compared with some general principles about
experiments, individuated looking at science and engineer-
ing (Sects. 3 and 4). The purpose of this section is to provide
a representative picture of how experimental activities are
carried out in mobile robot localization and mapping. We do
not aim at providing any comprehensive survey of the field.
Instead, we analyzed a number of papers, listed in Table 1,
that appear relevant to our goal. Some comments are worth
about the selection of the papers. We deem that the papers
we chose constitute a good sample, since they cover several
themes (including classical and more recent SLAM tech-
niques, outdoor SLAM, visual SLAM, scan matching), ex-
tend over almost two decades (from early 1990s to present),
have been written by authors from different continents, and
have been published in highly-respected journal and confer-
ences. Although, like in any selection, there is a subjective
bias, we do not consider it to be significant enough to harm
the representativeness of the global picture we provide.
For the sake of clarity, we organize the following pre-
sentation by discussing the purposes for which experiments
are reported in papers, the data sets on which experiments
are performed, and the measured quantities that are reported
to support claims. Since all these aspects are closely related
with each other and a linear presentation cannot account for
all the relationships, we try to highlight some of these rela-
tionships throughout the text.
2.1 Purposes of experiments
Experiments in mobile robot localization and mapping can
be devoted to demonstrate that the proposed system works,
to demonstrate that the proposed system works better than
other systems, to get insights on the behavior of a system
and on its limits of applicability, or, more frequently, to a
mixture of these goals. Let us consider these purposes in
more detail.
Localization and mapping algorithms are often hard to
characterize theoretically, for example through asymptotic
analysis of worst-case and average-case behavior. There-
fore, only few papers use analysis of computation com-
plexity to define machine-independent bounds on running
time (Leonard and Durrant-Whyte 1991). To demonstrate
that a system works, usually experiments are performed
to evaluate correctness and accuracy of the proposed algo-
rithms. Examples of papers in which experiments are de-
signed with this goal in mind are Davison (2003), Newman
et al. (2002, 2006), Paz et al. (2008).
Papers often compare performance of the proposed al-
gorithms with competing algorithms to demonstrate their
superiority (and that of the embedded ideas). This could
result in what Johnson (2002) calls a horse race paper,
where results are published for precisely defined, standard-
ized benchmarks to demonstrate the scientific value of an