The value of empirical evidence for practitioners and researchers

Abstract

The empirical software engineering research community has two general aims: 1. To understand how software is actually developed and maintained; and 2. To understand what improvements should be made to software development and maintenance, and how those improvements should be implemented. Empirical software engineering research, therefore, is about both contemplation and action. It is a discipline which attempts to understand phenomena whilst at the same time trying to change those very phenomena (in order to improve them). And it is a discipline that, by definition, promotes empirical evidence as the primary source of reliable knowledge for achieving these two general aims. While the research community promotes empirical evidence as the primary source of reliable knowledge, software practitioners do not seem to value empirical evidence in quite the same way; indeed, software practitioners seem to treat empirical evidence as one among a number of sources of knowledge (others being personal preferences and values, personal experience, the opinions of local experts, and local constraints) where the value of each of these kinds of knowledge varies from situation to situation: for example, in one situation (perhaps a CMM Level 4 company), a decision is made on the basis of empirical evidence; in another situation (a CMM Level 1 company), a decision is made on the basis of local expertise. As a research community we of course need to (further) strengthen our collaborations with industry, because industry is both the most appropriate source of empirical evidence for developing our understanding (the research communities' first general aim) and the intended target for action (the research communities' second general aim). We also need to be aware that industry is also the most appropriate source for these other kinds of knowledge; and that practitioners are not necessarily persuaded to collaborate and to change and to improve primarily because of empirically-based knowledge (particularly quantitative data). Practitioners' preferences for different kinds of knowledge are implicitly acknowledged in, for example, the design of the Capability Maturity Model (where the lower Levels of the model do not promote the collection and use of quantitative data) and more explicitly acknowledged by, as an alternative example, Evidence Based Software Engineering (derived from Evidence Based Medicine) which seeks to combine various kinds of knowledge to improve practitioners' decision-making about technology adoption. Given these circumstances, perhaps the most appropriate and helpful recommendations for improvement that the research community can offer to industry are heuristics (rules-of-thumb) based on the careful aggregation of different kinds of empirical and non-empirical evidence. © Springer-Verlag Berlin Heidelberg 2007.