A Systematic Review of Theory Use in Software Engineering Experiments Jo E. Hannay, Dag I.K. Sj��berg, Member, IEEE, and Tore Dyba, �� Member, IEEE Abstract���Empirically based theories are generally perceived as foundational to science. However, in many disciplines, the nature, role and even the necessity of theories remain matters for debate, particularly in young or practical disciplines such as software engineering. This article reports a systematic review of the explicit use of theory in a comprehensive set of 103 articles reporting experiments, from of a total of 5,453 articles published in major software engineering journals and conferences in the decade 1993- 2002. Of the 103 articles, 24 use a total of 40 theories in various ways to explain the cause-effect relationship(s) under investigation. The majority of these use theory in the experimental design to justify research questions and hypotheses, some use theory to provide post hoc explanations of their results, and a few test or modify theory. A third of the theories are proposed by authors of the reviewed articles. The interdisciplinary nature of the theories used is greater than that of research in software engineering in general. We found that theory use and awareness of theoretical issues are present, but that theory-driven research is, as yet, not a major issue in empirical software engineering. Several articles comment explicitly on the lack of relevant theory. We call for an increased awareness of the potential benefits of involving theory, when feasible. To support software engineering researchers who wish to use theory, we show which of the reviewed articles on which topics use which theories for what purposes, as well as details of the theories��� characteristics. Index Terms���Theory, experiments, research methodology, empirical software engineering. �� 1 INTRODUCTION THERE are many arguments in favor of theory use. Theories offer common conceptual frameworks that allow the structuring of knowledge in a concise and precise manner, thus facilitating the communication of ideas and knowledge. Their level of abstraction enables the general- ization of knowledge independently of a specific time and place [5], [66], [91], [95], [98]. Theory is the means through which one may generalize analytically [85], [103], thus enabling generalization from situations in which statistical generalization is not desirable or possible, such as from case studies [103], across populations [64], and indeed, often from experiments, especially those in social and behavioral sciences [85], with which experiments in empirical software engineering share essential features. Such arguments have been voiced in the software engineering community as well, e.g., [6], [27], [43], [57], [89]. A theory provides explanations and understanding in terms of basic concepts and underlying mechanisms, which constitute an important counterpart to knowledge of passing trends and their manifestations. When developing better software engineering technology for long-lived industrial needs, building theories is a means to go beyond the mere observation of phenomena and to try to under- stand why and how these phenomena occur. Thus, theories are of potential use to both researchers and practitioners. However, the usefulness of theories for software engineering is a subject of discussion, and the actual use of theory in empirical studies of software engineering is not well known. Decisions and discussions regarding issues of theory in empirical software engineer- ing must be founded not only on an understanding of what a theory is and how it can be useful, but also on knowledge of the actual use of theories. The main motivation of this article is to contribute to the latter by reviewing the use of explanatory theories for software engineering. Animportantundertakinginempiricalsoftwareengineer- ing is to determine what development technology to deploy andwhatdeveloperstouseinwhatsituations inotherwords, interest lies in comparing various technologies and skills and in determining their effects on software development. For this, the experiment is one choice of research method in that it may be applied more or less directly to make such comparisons and to measure such effects. According to Shadish et al., experiments are suitable for causal description, that is, ���describing the consequences attributable to deliberately varying a treatment��� [85, p. 9], but do not in themselves provide causal explanation, that is, ���clarifying the mechanisms through which and the condi- tions under which [the cause-effect relationship] holds��� [85, p. 9]. The desire for causal explanation motivates the development of theory. Thus, the focus of this review is on theories that are used to explain, in one way or another, the cause-effect relationships investigated by experiments. A challenge facing software engineering researchers who are considering using theories to support their research, is IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 33, NO. 2, FEBRUARY 2007 87 . J.E. Hannay and D.I.K. Sj��berg are with Simula Research Laboratory, Department of Software Engineering, Pb. 134, NO-1325 Lysaker, Norway. E-mail: {johannay, dagsj}@simula.no. . T. Dyba �� is with Simula Research Laboratory and with SINTEF ICT, Department of Software Engineering, Safety, and Security, NO-7465 Trondheim, Norway. E-mail: tore.dyba@sintef.no. Manuscript received 2 June 2006 revised 6 Sept. 2006 accepted 3 Nov. 2006 published online 28 Dec. 2006. Recommended for acceptance by D. Rombach. For information on obtaining reprints of this article, please send e-mail to: tse@computer.org, and reference IEEECS Log Number TSE-0124-0606. 0098-5589/07/$25.00 �� 2007 IEEE Published by the IEEE Computer Society