A Reference Framework for Utilization of Software Operation Knowledge
Henk van der Schuur, Slinger Jansen, Sjaak Brinkkemper
Department of Information and Computing Sciences
Utrecht University
Utrecht, The Netherlands
fh.schuur, s.jansen, s.brinkkemperg@cs.uu.nl
Abstract—Knowledge of in-the-field software operation is
a broad but ill-defined and fragmentarily supported subject
and it is unclear how software vendors can take advantage of
such knowledge. This paper introduces and defines software
operation knowledge to unify existing definitions, and presents
an empirically evaluated framework that is designed to aid
product software vendors in gaining insight in the potential
role of such knowledge in advancement of their products,
practices and processes. The results of extensive case studies
performed at three European software vendors show that if
used correctly, software operation knowledge enables vendors
to increase software quality and improve end-user experience.
However, case study results also illustrate that the state of
knowledge integration is still pragmatic and immature. Vendors
have to adapt their workflows, processes and tools to enable
structural software operation knowledge utilization.
Keywords-software performance, software quality, software
usage, software feedback, software process improvement
I. INTRODUCTION
Software vendors have recently begun discovering the
yields of software and end-user feedback. For example, by
implementing feedback reporting in its operating systems, a
large software vendor discovered that circa 50% of failures
are caused by one percent of software bugs [1]. If used
correctly, feedback enables software vendors to establish
how successful their products and services are at achieving
their goals in the field. These goals are dependent on soft-
ware end-users, and constitute aspects such as performance,
quality and usability.
With the increase of software complexity and ever higher
end-user expectations, advanced techniques are required to
monitor operations of software in the field. Common exam-
ples are crash reporting applications and service performance
monitoring tools. More exotic mechanisms exist, such as
‘software tomography’ [2] for monitoring specific aspects
of an application, end-user tracing for UI improvement,
as well as mechanisms for providing and delivering end-
user feedback. The software community has picked up
on the need for tools to support software and end-user
feedback concerns. Google, for example, has created the
Google Website Optimizer1, which enables website builders
to leverage end-user behavior by presenting end-users with
1http://www.google.com/websiteoptimizer/, verified 26/05/2010
different user interfaces and then measuring differences in
conversion rates, site effectiveness, visitor satisfaction, etc.
As another example, Mozilla has developed the Firefox Test
Pilot plug-in2 to get operation feedback and usage traces
from circa one percent of its end-users.
It remains unclear, however, how and to which extent
software and end-user feedback can be used to improve a
software vendor’s practices, processes and products. Sev-
eral research examples that focus on specific solutions and
domains can be found. The Skoll project [3] focuses on
user community-supported quality assurance of software
operating in large configuration spaces. Furthermore, the
GAMMA project [4] uses software tomography to gather
useful information from deployed software and focuses on
determining effective probe insertion locations. While these
examples show that research in this area is fragmented,
software and end-user feedback are generally used as main
data source.
An integrated view is needed that provides product soft-
ware vendors with insight in the potential role of such
feedback in advancement of their products, practices and
processes. The contribution of this paper is twofold:
 A definition is introduced to unify existing definitions
and uses of software feedback, as well as types of
knowledge emerging from in-the-field software operation
 A framework is presented that models the life cycle of
such knowledge as well as product software perspectives
from which processes of this life cycle can be perceived.
Both the definition and the framework are empirically
evaluated with a questionnaire and three investigative case
studies at European software vendors.
This paper continues with placing our work into context
and with the introduction of the software operation knowl-
edge definition (section II) and framework (section III). The
research evaluation approach is described in section IV.
Next, results of our empirical study are presented in sec-
tions V and VI. Finally, limitations of this research are dis-
cussed in section VII and research conclusions are presented
in section VIII.
2http://labs.mozilla.com/testpilot/, verified 26/05/2010