Preliminary Guidelines for Empirical Research
in Software Engineering
Barbara A. Kitchenham, Member, IEEE Computer Society,
Shari Lawrence Pfleeger, Member, IEEE, Lesley M. Pickard, Peter W. Jones,
David C. Hoaglin, Khaled El Emam, Member, IEEE Computer Society, and
Jarrett Rosenberg, Member, IEEE Computer Society
AbstractÐEmpirical software engineering research needs research guidelines to improve the research and reporting processes. We
propose a preliminary set of research guidelines aimed at stimulating discussion among software researchers. They are based on a
review of research guidelines developed for medical researchers and on our own experience in doing and reviewing software
engineering research. The guidelines are intended to assist researchers, reviewers, and meta-analysts in designing, conducting, and
evaluating empirical studies. Editorial boards of software engineering journals may wish to use our recommendations as a basis for
developing guidelines for reviewers and for framing policies for dealing with the design, data collection, and analysis and reporting of
empirical studies.
Index TermsÐEmpirical software research, research guidelines, statistical mistakes.
æ
1INTRODUCTION
W
E have spent many years both undertaking empirical
studies in software engineering ourselves and review-
ing reports of empirical studies submitted to journals or
presented as postgraduate theses or dissertations. In our
view, the standard of empirical software engineering
research is poor. This includes case studies, surveys, and
formal experiments, whether observed in the field or in a
laboratory or classroom. This statement is not a criticism of
software researchers in particular; many applied disciplines
have problems performing empirical studies. For example,
Yancey [50] found many articles in the American Journal of
Surgery (1987 and 1988) with ªmethodologic errors so
serious as to render invalid the conclusions of the authors.º
McGuigan [31] reviewed 164 papers that included numerical
results that were published in the British Journal of Psychiatry
in 1993 and found that 40 percent of them had statistical
errors. When Welch and Gabbe [48] reviewed clinical articles
in six issues of the American Journal of Obstetrics, they found
more than half the studies impossible to assess because the
statistical techniques used were not reported in sufficient
detail. Furthermore, nearly one third of the articles con-
tained inappropriate uses of statistics. If researchers have
difficulty in a discipline such as medicine, which has a rich
history of empirical research, it is hardly surprising that
software engineering researchers have problems.
In a previous investigation of the use of meta-analysis in
software engineering [34], three of us identified the need to
assess the quality of the individual studies included in a
meta-analysis. In this paper, we extend those ideas to
discuss several guidelines that can be used both to improve
the quality of on-going and proposed empirical studies and
to encourage critical assessment of existing studies. We
believe that adoption of such guidelines will not only
improve the quality of individual studies but will also
increase the likelihood that we can use meta-analysis to
combine the results of related studies. The guidelines
presented in this paper are a first attempt to formulate a
set of guidelines. There needs to be a wider debate before
the software engineering research community can develop
and agree on definitive guidelines.
Before we describe our guidelines, it may be helpful to
you to understand who we are and how we developed
these guidelines. Kitchenham, Pickard, Pfleeger, and El-
Emam are software engineering researchers with back-
grounds in statistics as well as computer science. We
regularly review papers and dissertations, and we often
participate in empirical research. Rosenberg is a statistician
who applies statistical methods to software engineering
problems. Jones is a medical statistician with experience in
developing standards for improving medical research
studies. Hoaglin is a statistician who has long been
interested in software and computing. He reviewed eight
papers published in Transactions on Software Engineering in
the last few years. These papers were not chosen at random.
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 28, NO. 8, AUGUST 2002 721
. B.A. Kitchenham and L.M. Pickard are with the Computer Science
Department, Keele University, Keele, Staffordshire ST5 5BG, UK.
E-mail: {barbara, lesley}@cs.keele.ac.uk.
. S.L. Pfleeger is with the RAND Corporation, 1200 South Hayes St.,
Arlington, VA 22202-5050. E-mail: shari_pfleeger@rand.org.
. P.W. Jones is with the Mathematics Department, Keele University, Keele,
Staffordshire ST5 5BG, UK. E-mail: maa11@maths.keele.ac.uk.
. D.C. Hoaglin is with Abt Associates Inc., 55 Wheeler Street, Cambridge,
MA 02138. E-mail: Dave_Hoaglin@abtassoc.com.
. K. El Emam is with the National Research Council of Canada Institute for
Information Technology, Montreal Road, Building M-50 Ottowa, Ontario,
K1J 8H5 Canada. E-mail: khaled.el-emam@seg-nrc.org.
. J. Rosenberg is with Sun Microsystems, San Antonio Road, MS MPK17-
307 Palo Alto, CA 94303. E-mail: Rosenberg@eng.sun.com.
Manuscript received 30 Dec. 1999; revised 26 Jan. 2001; accepted 23 Apr.
2001.
Recommended for acceptance by M. Jazayeri.
For information on obtaining reprints of this article, please send e-mail to:
tse@computer.org, and reference IEEECS Log Number 111161.
0098-5589/02/$17.00 ß 2002 IEEE