Replicating Software Engineering Experiments:
Addressing the Tacit Knowledge Problem


Forrest Shull,
Fraunhofer Center –
Maryland
fshull@fc-md.umd.edu
Victor Basili
Fraunhofer Center –
Maryland and
University of Maryland
basili@fc-md.umd.edu
Jeffrey Carver
University of Maryland
carver@cs.umd.edu
José C. Maldonado
ICMC-USP
jcmaldon@icmc.sc.usp.br

Guilherme Horta Travassos
COPPE/UFRJ
ght@cos.ufrj.br
Manoel Mendonça
UNIFACS
mgmn@unifacs.br
Sandra Fabbri
UFSCar
sfabbri@dc.ufscar.br



Abstract
Recently the awareness of the importance of
replicating studies has been growing in the empirical
software engineering community. The results of any one
study cannot simply be extrapolated to all environments
because there are many uncontrollable sources of
variation between different environments.
In our work, we have reasoned that the availability of
laboratory packages for experiments can encourage better
replications and complementary studies. However, even
with effectively specified laboratory packages, transfer of
experimental know-how can still be difficult. A
cooperation between Brazilian and American researchers
addressing effective running of replications was formed in
1999. One of the specific issues being addressed is the
problem of transferring tacit knowledge.
We discuss what we learned about the tacit knowledge
transfer problem and the evolution of laboratory packages
in the description of a replication performed in Brazil
using a PBR laboratory package.; also how further issues
will be addressed.

1. Introduction
In the past few years, there has been a growing
awareness in the empirical software engineering
community of the importance of replicating studies (e.g.
[5], [11], [9], [2], [13], [15]). Researchers realize that the
true goal of empirical research should be not the running
of individual studies but developing a better understanding
of software development, the cost and benefits of various
techniques and at the very end consolidating a body of
knowledge and establishing software development models.
Too many uncontrollable sources of variation exist from
one environment to another for the results of any study, no
matter how well run, to be extrapolated to all possible
software development environments. Most researchers
accept that no one study on a technology should be
considered definitive.
A result of this realization is an increased
commitment to run more studies in a variety of
environments. Replication in different environments is an
important characteristic of any laboratory science. It is the
basis for credibility and learning. Complementary,
replicated studies allow researchers to combine knowledge
directly or via some form of meta-analysis. Since
intervening factors and threats to validity can almost never
be completely ruled out of a study, complementary studies
also allow more robust conclusions to be drawn when
related studies can address one another’s weak points. In
software engineering, this enables us to build a body of
knowledge about families of related techniques and basic
principles of software development.
For clarity a brief working definition of a replication
will be given here. While in many contexts the term
replication implies repeating a study without making any
changes, this definition is too narrow for our purposes. In
this work we will consider a replication to be a study that
is run, based on the design and results of a previous study,
whose goal is to either verify or broaden the applicability
of the results of the initial study. For example, the type of
replication where the same exact study is run could be
used to verify results of an original study. On the other
hand, if a researcher wished to explore the applicability of
the results in a different context, then the design of the
original study may be slightly modified but still
considered a replication.
In our own work, we have reasoned that better
replications and complementary studies can be encouraged
Proceedings of the 2002 International Symposium on Empirical Software Engineering (ISESE’02)
0-7695-1796-X/02 $17.00 © 2002 IEEE