Customer Configuration Updating
in a
Software Supply Network
PROEFSCHRIFT
ter verkrijging van de graad van doctor
aan de Universiteit Utrecht op gezag van
de rector magnificus, prof. dr. W. H. Gispen,
ingevolge het besluit van het college
voor promoties in het openbaar te verdedigen op
8 oktober 2007 des ochtends te 10.30 uur
door
Slinger Remy Lokien Jansen
geboren op 1 juni 1980,
te Rotterdam

Promotoren: Prof. Dr. S. Brinkkemper
Prof. Dr. P. Klint
Dit onderzoek is deel van het Deliver project, betaald door NWO/Jacquard,
projectnummer 638.001.202.

The work in this thesis has been carried out at Utrecht University and the Centrum
voor Wiskunde en Informatica (CWI) in Amsterdam. The research reported in this
thesis has been carried out under the auspices of SIKS, the Dutch Research School for
Information and Knowledge Systems. SIKS Dissertation Series No. 2007-20.
iii

Acknowledgements
Before being thrown into the depths of customer configuration updating I would like
to acknowledge the people who have been there for me the last four years. First and
foremost I would like to thank the girl going through all the ups and downs, the down
and outs, and the high and low points of a researcher. If you will stand by me for the
rest of my life as you have done these years, Merel, I am guaranteed a happy future. My
family is next, my mother Carrie, my father Geertje, and my sister Blixa, thanks for the
trips, the meals, the christmases, and the overall entertainment during those laborious
times. I specifically have to thank Blixa for keeping me young (and giddy).
I must thank Vedran, even after his treacherous move to Venlo, for providing me
with at least 50% of the brainpower for this thesis. As our Judo trainer once said: “you
guys look as if you know just that little more about life.” Every day I hope he is right,
and every day I hope we get to have our American style Barbeque when we are 45.
I would like to thank our Matriarch Doortje Wisse and all my aunts and uncle,
for helping me out when stuff went horribly wrong (Dorotee, Birgitta, Julia, Yvje,
Clemens, and the rest), their partners who put life into perspective when necessary
(Ab), and of course Prof. Dr. Marian, who will be a motivating force for the rest of
my career. Furthermore, I owe great thanks to Diana Emmink, Adriaan, Harald, and
Rogier van Geest, for being the best in-laws a guy can wish for.
My friends as well as colleagues have helped loads, either personally or
professionally. I am of course speaking of my esteemed friends Tijs, Magiel, Jurgen,
Eva, and Inge. Furthermore I would like to thank Arie, Rob, Gerco, Lidwien, Ilja,
Remko, Ronald, and Johan. Also, I must thank my dear friends and colleagues
overseas, being Sue Black, Anthony Finkelstein, Andy Maule, and all others in
London. Finally I want to thank those that have taught me life’s lessons in no particular
order, Karel Philipsen, Miss Ewings, Marko Steenbergen, (soon to be Professors)
Kosters and Hogeboom, Aad van Polanen, Meneer van de Broek, Rene de Jong, dokter
Meijer, Harry Ottink, Theo Ham, and Erik Bulk. Finally I’d like to thank Sjaak
Brinkkemper and Paul Klint, for being mentors, friends, colleagues, and bosses on-
demand.

Contents
Preface v
Contents vii
I Introduction to this Thesis 1
1 Introduction 3
1.1 Research Area . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.1.1 Product Software . . . . . . . . . . . . . . . . . . . . . . . . 4
1.1.2 Product Management Research Perspectives . . . . . . . . . . 5
1.1.3 Customer Configuration Updating . . . . . . . . . . . . . . . 5
1.1.4 Product Software Development and Maintenance . . . . . . . 7
1.1.5 Overview of Concepts . . . . . . . . . . . . . . . . . . . . . 9
1.2 Research Setting and Industrial Embedding . . . . . . . . . . . . . . 10
1.2.1 Academic Research Centers . . . . . . . . . . . . . . . . . . 10
1.2.2 Platform for Product Software . . . . . . . . . . . . . . . . . 11
1.2.3 Industry Context and Applicability . . . . . . . . . . . . . . . 12
1.3 Research Description . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.3.1 Research Questions . . . . . . . . . . . . . . . . . . . . . . . 13
1.3.2 Research Approach . . . . . . . . . . . . . . . . . . . . . . . 15
1.3.3 Research Methods . . . . . . . . . . . . . . . . . . . . . . . 16
1.4 List of Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
1.5 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
II Current Practice 21
2 Definition and Validation of Customer Configuration Updating 23
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.2 Process Areas for Customer Configuration Updating . . . . . . . . . 25
2.2.1 Release Process Area . . . . . . . . . . . . . . . . . . . . . . 26
2.2.2 Delivery Process Area . . . . . . . . . . . . . . . . . . . . . 27
2.2.3 Deployment Process Area . . . . . . . . . . . . . . . . . . . 27
2.2.4 Activation and Usage Process Area . . . . . . . . . . . . . . 29
vii

CONTENTS
2.3 The Cases and their Key Practices . . . . . . . . . . . . . . . . . . . 29
2.3.1 Case Study Approach . . . . . . . . . . . . . . . . . . . . . . 30
2.3.2 Hospital Information Management System . . . . . . . . . . 31
2.3.3 On-line ERP Information Portal for Large Businesses . . . . . 31
2.3.4 Content Management System . . . . . . . . . . . . . . . . . 32
2.3.5 Providing a Counter Service On-Line . . . . . . . . . . . . . 32
2.3.6 Facility Management System . . . . . . . . . . . . . . . . . . 33
2.3.7 CAD plug-in for Building Design . . . . . . . . . . . . . . . 33
2.3.8 Mozilla Firefox . . . . . . . . . . . . . . . . . . . . . . . . . 34
2.3.9 Apache’s HTTP Server . . . . . . . . . . . . . . . . . . . . . 34
2.4 Evaluation of the Process Areas and Features . . . . . . . . . . . . . 34
2.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.6 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3 A Case Study in Mass Market ERP Software 45
3.1 Integrated Development and Maintenance . . . . . . . . . . . . . . . 45
3.2 Research Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.2.1 Problem Overview . . . . . . . . . . . . . . . . . . . . . . . 47
3.2.2 Exact Software . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.2.3 The Case Study . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.3 The SKB and its use within ES . . . . . . . . . . . . . . . . . . . . . 51
3.3.1 SCM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
3.3.2 PDM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
3.3.3 CRM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.4 Maintenance and the SKB . . . . . . . . . . . . . . . . . . . . . . . 56
3.4.1 Maintenance at the Development Site . . . . . . . . . . . . . 56
3.4.2 Maintenance at the Customer . . . . . . . . . . . . . . . . . . 58
3.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
3.6 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
3.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
4 A Benchmark Survey into the Customer Configuration Processes 65
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
4.2 Customer Configuration Updating . . . . . . . . . . . . . . . . . . . 67
4.2.1 Processes and Practices . . . . . . . . . . . . . . . . . . . . . 67
4.3 Research Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
4.3.1 Hypotheses . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
4.3.2 Approach and Survey Design . . . . . . . . . . . . . . . . . 70
4.3.3 Sample Selection . . . . . . . . . . . . . . . . . . . . . . . . 70
4.3.4 Survey Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
4.3.5 Benchmark Survey . . . . . . . . . . . . . . . . . . . . . . . 72
4.3.6 Validity Threats . . . . . . . . . . . . . . . . . . . . . . . . . 72
4.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
4.4.1 Respondents . . . . . . . . . . . . . . . . . . . . . . . . . . 73
4.4.2 Hypotheses Results . . . . . . . . . . . . . . . . . . . . . . . 78
4.4.3 Open Questions . . . . . . . . . . . . . . . . . . . . . . . . . 78
viii

CONTENTS
4.4.4 Suggestions for Customer Configuration Updating (CCU)
Improvement . . . . . . . . . . . . . . . . . . . . . . . . . . 79
4.4.5 Exploring Relationships . . . . . . . . . . . . . . . . . . . . 80
4.5 CCU Practice Results . . . . . . . . . . . . . . . . . . . . . . . . . . 81
4.5.1 Release . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
4.5.2 Delivery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
4.5.3 Deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
4.5.4 Usage and Activation . . . . . . . . . . . . . . . . . . . . . . 83
4.6 Conclusions and Discussion . . . . . . . . . . . . . . . . . . . . . . 84
4.7 The Survey and the CCU Relationship Tables . . . . . . . . . . . . . 86
III Tool Support for CCU 95
5 A Process Framework and Typology for Software Product Updaters 97
5.1 Product Updating . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
5.2 The Product Software Updating Process . . . . . . . . . . . . . . . . 98
5.2.1 Update Process Framework . . . . . . . . . . . . . . . . . . 98
5.2.2 A Typology for Product Updaters . . . . . . . . . . . . . . . 100
5.2.3 Evaluation of Update Process Coverage . . . . . . . . . . . . 101
5.2.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
5.3 Delivery and Deployment . . . . . . . . . . . . . . . . . . . . . . . . 103
5.3.1 Delivery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
5.3.2 Deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
5.3.3 Evaluation of Delivery and Deployment . . . . . . . . . . . . 105
5.4 Discussion and Future Work . . . . . . . . . . . . . . . . . . . . . . 107
5.4.1 Typology . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
5.4.2 Delivery and Deployment . . . . . . . . . . . . . . . . . . . 109
5.4.3 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . 109
5.4.4 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . 109
5.4.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
5.5 Short Description of Update Technologies Used . . . . . . . . . . . . 110
6 Modelling Deployment using Feature Descriptions and State Models 113
6.1 Component Deployment Matters . . . . . . . . . . . . . . . . . . . . 113
6.2 Component Descriptions . . . . . . . . . . . . . . . . . . . . . . . . 115
6.2.1 Component States and Instantiations . . . . . . . . . . . . . . 117
6.2.2 Feature Diagrams . . . . . . . . . . . . . . . . . . . . . . . . 120
6.3 Instantiation Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
6.3.1 Example 1: Instantiating the Pop3 Component . . . . . . . . 122
6.3.2 Example 2: Instantiating the e-Mail Client . . . . . . . . . . . 126
6.3.3 Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
6.4 Component Knowledge Management . . . . . . . . . . . . . . . . . 131
6.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
6.6 Conclusions and Future Work . . . . . . . . . . . . . . . . . . . . . 134
ix

CONTENTS
V Conclusion 181
10 Conclusion 183
10.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
10.2 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
10.2.1 Research Methods . . . . . . . . . . . . . . . . . . . . . . . 186
10.3 Chapter Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . 186
10.4 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
Bibliography 189
Publication List 201
Summary 205
Nederlandse Samenvatting: Actualiseren van Klantconfiguraties via een
Softwareleverantienetwerk 209
Resume 215
xi

Introduction CHAPTER 1
customer configuration updating. The first contribution is a customer configuration
updating process improvement model that enables product software vendors to make
strategic improvement decisions. The second contribution is a tool evaluation method
that enables customer configuration updating support tool builders to establish what
features are required for such tools. The third contribution is a tool infrastructure for
software knowledge delivery, enabling product software vendors to share knowledge
with their end-users in a software supply network. Finally, the fourth contribution
is a tool that facilitates correct evolution between configurations of components
while keeping complexity within manageable borders. The importance of customer
configuration updating is uncovered by nine case studies of product software and a
survey, showing that customer configuration updating improvements have had a large
influence on product success.
1.1 Research Area
1.1.1 Product Software
Product software is a packaged configuration of software components or a software-
based service, with auxiliary material, which is released for and traded in a specific
market [132]. In contrast, product software is different from embedded software
because product software is sold separately from the hardware on which it will be
installed. Furthermore, it is different from customly built software for one customer, in
that because product software is delivered to a large number of customers and deployed
on a wide variation of hardware components.
Some examples of product software are games, components-off-the-shelf, database
management systems, and ERP packages:
 TomTom [129], the manufacturer of navigational software and embedded
devices, develops the software product TomTom navigator. This product is a
software product because it can be deployed on a large range of devices (PC,
PDA, and embedded devices) and is sold separately. TomTom has demonstrated
an explosive growth throughout the last years and is now active in eighteen
countries.
 Zylom games [27] is a manufacturer and service provider of games delivered
over the Internet. Their target customers are casual gamers who wish to play a
quick game. For a small amount of money gamers can play all games without
limitations. Furthermore, gamers can participate in international tournaments.
This Dutch founded company currently has 10 million subscribed casual gamers.
 Exact Software [26] is a manufacturer of product software and one of the largest
product software vendors in the Netherlands. Their ERP products, making
bookkeeping for small to medium companies simple, are being sold all over the
world. Currently their customer base exceeds 160,000 customers.
Besides illustrating what product software entails, these three Dutch examples show
that product software vendors can be extremely successful. Furthermore, these three
4

SECTION 1.1 Research Area
provide only a minor part of the 1.6 billion euros in exports from the Netherlands.
Besides being a booming business, the OECD [44] claims that product software is one
of the most rapidly growing sectors in OECD countries.
1.1.2 Product Management Research Perspectives
Product software management can be seen from three perspectives [132], being the
social perspective, the company perspective, and the development perspective. The
social perspective views all external factors that influence a product software vendor,
such as laws and regulations and the economy. The company perspective concerns
all non-development processes, such as marketing, sales, quality control, etc. Finally,
the development perspective concerns all processes that eventually produce software
products that can readily be deployed at the customers. The product software research
framework is displayed in figure 1.1 [132].
The focus of this thesis is on the development perspective, and more specifically
the development, release, delivery, and deployment of product software (found in
the bottom right corner of figure 1.1). The development process determines how
bugs are resolved, feature requests are satisfied, the development methodology,
etc. Furthermore, the release process determines how and when new releases are
published, keeping in mind the releases that are already installed at customers.
The delivery process concerns the delivery of software artefacts from customer to
vendor, again keeping in mind the releases that are deployed at customer sites. The
deployment process concerns not only the technical deployment of products, but also
the implementation of these products into the customers’ organization.
1.1.3 Customer Configuration Updating
One area specific to product software vendors is that they have to release, deliver,
and deploy their products on a wide range of systems, for a wide range of customers,
in many variations. Furthermore, these applications constantly evolve, introducing
versioning problems. An increasingly important part of product software development
thus is CCU. CCU is the combination of the vendor side release process, the product
or update delivery process, the customer side deployment process, and the activation
and usage processes. Product software vendors encounter particular problems when
trying to improve these processes, because vendors have to deal with multiple revisions,
variable features, different deployment environments and architectures, different
customers, different distribution media, and dependencies on external products. Also,
there are not many tools available supporting the delivery and deployment of software
product releases that are generic enough to accomplish these tasks for any product. For
a complete description of CCU we direct the reader to section 2.2.
This thesis does not stand alone in its attempt to improve CCU for product
software vendors. The work on evaluating product updaters is largely based on
an earlier evaluation model provided by Carzaniga et al [21]. Furthermore, the
entrepreneurial aspects of this work were inspired by Xu and Brinkkemper’s work on
product software [132]. The tool Pheme was largely inspired by the Software Dock [52]
and can be considered a next generation of it.
5

Introduction CHAPTER 1
Figure 1.1: Product Software Research Framework
6

SECTION 1.1 Research Area
An integrated view of CCU is rarely presented in literature. There is, however, a
large body of knowledge about the separate CCU processes.
Release management is divided into two areas, being release support tools and
release planning. Release planning [122, 6, 20] is often seen as a wicked problem.
This thesis only touches the surface of release planning, and attempts to find a generic
process description for release planning, instead of detailed solutions for feature and
bug prioritization [92] and solutions that describe new release methods [124]. The
solutions presented focus on software products as blank artefacts. With respect to
release tools this thesis does present a fairly simple release tool. However, the
release tool presented in this thesis assumes the product is ready for delivery, whereas
others [125], do not.
In regards to software delivery no research specifically addresses delivery
of product software knowledge, with the exception of the works of Farbey and
Finkelstein [42]. Other works on software delivery [61] focus more on the technical
problems of delivery, such as trying to reduce overhead and delivery cost.
The work on deployment of components in multidimensional configurations could
not have been carried out without the work of van der Storm revealing that these
configuration spaces can be reduced to binary decision trees [123]. Other works on
deployment and updating generally only look at technical aspects of deployment. An
inspirational example of this is Nix [35], a tool that ensures complete and correct
deployment by storing a customer’s complete component configuration in a versioned
repository. This tool elegantly updates component configurations but does not at all
focus on the organizational aspects of deployment. Another example is the thesis of
Ajmani [2] who has developed a theoretical model for run-time updates of distributed
systems. Though this system is highly advanced and elegantly performs runtime
updates, it does not describe any of the organizational deployment issues, such as
dependencies on other applications and evolution of customer specific solutions.
With regards to usage and activation a research trend can be observed that
focuses more on software quality and profiling of software products in their operational
environments. Some examples of research projects focusing on usage and error
feedback are Skoll project [77], EDEM [56], and GAMMA project [96], which uses
a technique called software tomography to get valuable information from running
deployed software. The techniques proposed by Elbaum and Diep [41] have been
inspirational to this thesis, however, these techniques focus solely on where a software
application must be probed to gather information on a deployed software application.
1.1.4 Product Software Development and Maintenance
Product software development is the activity of development, modification, reuse, re-
engineering, maintenance, or any other activities resulting in packaged configurations
of software components or software-based services. Product software development
shares many processes and concepts with software engineering, although software
engineering is all encompassing for the building of software, whereas product software
development specifically looks at software products released for a market.
One of the main drivers for this research is the need to deliver working software
quicker and to deliver software of higher quality. Customers expect better software and
7

Introduction CHAPTER 1
shorter iterations between releases. Software developers are able to satisfy this demand
by constantly testing, releasing, and generating more code. This type of software
development is known as agile software development [10]. Some of the development
methodologies originating from the agile camp are Extreme Programming [11] and
Scrum [105] and are suitable methods to apply to product software development.
As product software vendors grow larger they find that product software knowledge
management is a critical success factor. Sales personnel must know when the next
release is coming out, what features are being developed at the moment, and what they
can and cannot show customers. Software developers need to know when the next
release is due, what customer concerns are, and what requirements deserve priority.
Also, helpdesk personnel need to know what issues are being fixed, for which issues
workarounds are available, etc. These are just some of the examples of knowledge that
needs to be managed by product software vendors.
Product software knowledge management is the driving factor behind high quality
CCU processes. For these processes knowledge is required about the product,
such as product features, relationships to third-party products, and product licensing
possibilities. Not only must this knowledge be managed internally, such that
stakeholders can acquire this knowledge at any time, but this knowledge also
needs to be shared with customers, third-party component providers and third-party
implementation partners. For the CCU processes, different types of knowledge are
required.
To release software a vendor needs to know when a product release will be finished,
what product releases have already been released, the bill of materials for a release, and
a products relationship to other products. Furthermore, a product software vendor must
clearly define its debug and release policies, which together define release planning.
Also, a vendor must define policies on how the release policy is shared with external
parties.
To deliver software a vendor needs to determine what customers already have, how
product artefacts are transported to the customer, and how often. A vendor needs to
clearly specify how product releases are published. Furthermore, a vendor must deliver
products to customers when they want products, not when the vendor feels like it. For
the delivery process the customer also plays a part in knowledge management, because
they decide when and how product usage and error feedback is delivered to the vendor.
To deploy software a vendor must make sure that all product requirements and
dependencies have been made explicit, preferably in a human and computer readable
format, and that this knowledge is at the customer side at the time of deployment.
Furthermore, the vendor needs to know exactly how a customer’s configuration can
be updated to contain a vendor’s new features and products. Also, vendors potentially
need to train customers and keep them up-to-date about product developments.
Finally, the usage and activation process are for a large part the customer’s
responsibility. They decide when a license is used to activate a product, when
knowledge about the configuration is delivered to the vendor, and how bug reports
and feature requests are sent to the vendor.
8

Introduction CHAPTER 1
1.2.3 Industry Context and Applicability
Clearly this research has been strongly embedded in the product software industry.
There are a number of reasons for this.
The CCU processes are described to a limited extent in common literature, resulting
in a lack of process descriptions, from which both industry and academics could profit.
Furthermore, because product software vendors were the first to encounter problems
with CCU manageability, academia has not yet developed equally advanced solutions.
The industrial partners have also provided us with a number of critical success factors
for the implementation of these processes.
The second reason why this work is strongly embedded in industry is that the
industrial partners enabled tool evaluation from a different perspective. The product
software vendors provided us with reviews and change requests for the Pheme
prototype that is presented in chapter 7. Also, the product software vendors enabled
evaluation of proprietary CCU support tools, providing a richer dataset. These
proprietary support tools are discussed in chapter 5.
The third reason is that the product software vendors have enabled evaluation of
the process models and improvement propositions. Each case study was finalized
with an advisory report indicating strengths and weaknesses of their processes and
that proposed a number of improvements, based on the process models. These
improvements were praised and criticized by the product software vendors.
This research is, besides being highly relevant academically, a contribution to the
product software industry. Product software vendors can use this thesis as a set of
guidelines when designing CCU support tools and implementing these tools and CCU
processes into their organizations.
1.3 Research Description
Product software vendors are impeded in their growth because they cannot adequately
share knowledge with customers about products. Furthermore, these product software
organizations increasingly cooperate in complex software supply networks, where one
software vendor is the customer of another. These situations require an even larger
amount of knowledge to be shared over a web of vendors and customers. To improve
this situation software vendors need to know what knowledge they need to share with
customers and how to share it.
Knowledge about products is shared when vendors release, deliver, deploy and
when customers use a software product. These CCU processes have not yet before
been properly modeled and evaluated. Furthermore, there are few tools that support
large parts of the CCU processes. When software vendors attempt to improve CCU
three problems become apparent:
 There are no adequate process descriptions for CCU,
 There is a lack of tools to support CCU,
 Each software vendor spends a lot of time automating CCU tasks, even though
these tasks are similar for all software vendors.
12

Introduction CHAPTER 1
state of the art provides us with typical problems and measuring tools to evaluate the
presented research.
SQ1.2: What is the state-of-the-practice of customer configuration updating for
product software vendors? The state of the practice provides us with a view of how
CCU is currently implemented at most product software vendors. This enables us to
compare “ideal” theoretical solutions with solutions that actually work in practice.
Furthermore, descriptions from the state of the practice enable us to test research
results in practical settings and the model descriptions from sub question 1. By looking
at product software vendors’ best practices can be uncovered for the product software
industry and software engineering. These best practices can be used to inform both the
scientific community and other product software vendors on what best practices exist
regarding knowledge sharing.
SQ1.3: What parts of the CCU processes are currently supported by product
update tools? Many practical problems from CCU processes are caused by the absence
of adequate tools. Before making such judgment and acting upon that by building
new prototype tools, however, an inventory is needed of what features are currently
provided by product software update tools. Such an inventory will uncover gaps in
CCU processes and tools, and enable product software vendors to make informed
decisions in make-or-buy situations.
RQ2: Can customer configuration updating be improved by explicitly
managing and sharing knowledge about software products?
The scientific motivation for the research question is to develop processes and tools
for the fully automatic consistency checking and web-based deployment, upgrading
and integration of product software. The goal of the research is to provide academics
and practitioners with comprehensive process models and tools that support the CCU
processes, enabling both groups to improve the area of product software engineering,
maintenance, and management. The economic motivation for the research question
is to strengthen the manufacturing and deployment of product software in the
Netherlands even more.
SQ2.1: What aspects of customer configuration updating can be improved by
explicitly managing and sharing customer configuration updating knowledge and can
these improvements be measured? The first step in answering this research question is
to discover which parts of the product software development process are potentially
improved. To do so a number of measures need to be developed to quantify these
improvements. Such measurements enable the explicit definition of the contribution of
this research.
SQ2.2: Are product software vendors who explicitly manage customer
configuration updating knowledge more successful? A first insight into what
14

SECTION 1.3 Research Description
areas of product software development can be improved is provided by examples of
product software vendors who are more successful because they explicitly manage
software knowledge. Detailed descriptions of how these processes are improved and
what type of knowledge is being managed and distributed in complex networks of
product software vendors and customers enable further validation of the hypothesis
that explicit knowledge sharing contributes to product software development.
SQ2.3: What functionality is required from tools managing and reusing customer
configuration updating knowledge to support product software development and the
customer configuration updating processes? Product software vendors can only
explicitly manage product software knowledge with tools that can store software facts
(in both a computer and human readable format) due to the large amount of data.
To build such tools an inventory needs to be made of software knowledge that can
potentially be managed by such tools. Furthermore, there needs to be a demand for
such tools from a business perspective. Finally, the tools need to be evaluated in
different settings to establish their contribution.
1.3.2 Research Approach
There are three views of CCU, being the practice view, the tool view, and the process
view (see figure 1.2). The practice view concerns how CCU is applied in practice
by product software vendors and open source products. The tool view looks at
CCU support tools and their specific characteristics supporting the CCU processes.
The process view concerns CCU process design and process models enabling any
stakeholder of a software product to evaluate its CCU processes.
To answer the first research question about the current state of the practice of CCU,
two studies were conducted; one through case studies at product software vendors and
one through tool evaluations found in literature and used by the case study subjects.
These case studies and tool evaluations enabled the forming of hypotheses about the
Dutch product software industry, and about CCU support tools. It was soon found
that the product software vendors encountered problems while implementing CCU
processes and that the tools did not provide all features required by these product
software vendors. This was confirmed when the tools and cases were evaluated using
two specially designed evaluation models.
The CCU evaluation model was used to establish whether product software vendors
can be compared and whether their CCU processes are a source of new problems and
solutions. The CCU evaluation model enabled the forming of hypotheses about the
Dutch product software industry. These hypotheses were then tested using the CCU
survey. We then used the survey to establish empirical evidence for further theory
building.
The CCU support tool evaluation model was used to establish what features are
usually missing in these tools, what processes still provide technical challenges, and
how these tools compare to each other. Some missing features also came from
demands made by the product software industry. To improve the product software
15

Introduction CHAPTER 1
Figure 1.2: CCU Research Approach
CCU processes two tools were developed. These tools were evaluated in two different
scenarios. The tools and evaluation enabled further theory building.
1.3.3 Research Methods
This research has been conducted as a mixed method multi-theory study of CCU
processes. The methods used are case studies, survey research, design research, tool
evaluation, and prototype evaluation. The applied research methods per chapter and
publication are shown in table 1.1.
Chapter Publication Case Survey Design Tool Prototype
study research evaluation evaluation
2 [143, 139, 144] X X
3 [141, 148] X X X
4 [146] X X
5 [147] X X
6 [142] X X
7 [149] X X X
8 [145] X
9 [151, 152] X X
Table 1.1: Applied Research Methods
Case Studies - Nine case studies have been conducted applying the case study
method developed by Yin [133]. The case studies have contributed to defining the
problem area, defining the state of the practice, and finding tools in the field. In chapter
16

SECTION 1.5 List of Acronyms
3 one of these cases is described in further detail. Also, from these nine case studies
ten misconceptions about customer configuration updating are highlighted and clarified
using cost/value evaluation in chapter 8. In chapter 9 a case study is used to evaluate a
modeling method for software supply networks. The case studies were performed with
care and rigor, using Yin’s case study method and guidelines [133]. To avoid common
pitfalls we have used guidelines and pointers from Kitchenham and Flyvbjerg [69, 43].
Survey Research - The case studies, though useful for initial investigation and
finding new phenomena, did not enable full generalization of the conclusions drawn
about product software vendors in the Netherlands. To counter this, a survey was
held among 74 product software vendors in the Netherlands. Their results enabled
generalization of our conclusions and provide a clear image of the CCU processes of
Dutch product software vendors.
Design Research - Some of the conducted research is design research [118], where
solutions are designed and evaluated in different settings. Some examples are the
CCU model in chapter 2, the product development cycle time model and the Pheme
Knowledge Delivery Infrastructure described in chapter 7, the software deployment
modeling technique and tool in chapter 6, and the software supply network modeling
technique presented in chapter 9.
Tool Evaluation - Tool evaluation in this research is seen as a specific case
study method to determine properties and features of CCU processes support tools.
To establish what tools are appropriate to support (C-)CCU processes, tools were
evaluated by testing and applying them to real-life examples in chapter 7. The result of
this research is a list of features that are currently insufficiently provided by (C-)CCU
support tools.
Prototype Evaluation - Two tools were built to demonstrate feasibility and to
establish that they actually improved (C-)CCU. These tools were evaluated the same
way as the other academic and commercial tools in chapters 6 and 7.
1.4 List of Acronyms
PDM Product Data Management
SCM Software Configuration Management
CRM Customer Relationship Management
SKB Software Knowledge Base
CCU Customer Configuration Updating
C-CCU Continuous Customer Configuration Updating
SSN Software Supply Network
COTS Components off the Shelf
ASP Application Service Provider
17

C H A P T E R 2
Turning the Ugly Duckling
into a Swan
For software vendors the processes of release, delivery, and deployment
to customers are inherently complex. However, software vendors could
greatly improve their product quality and quality of service by applying
a model that focuses on customer interaction if such a model were
available. This chapter presents a model for Customer Configuration
Updating (CCU) that can evaluate the practices of a software vendor in
these processes. Nine extensive case studies of medium to large product
software vendors are presented and evaluated using the model, thereby
uncovering issues in their release, delivery, and deployment processes.
Finally, organizational and architectural changes are proposed to increase
quality of service and product quality for software vendors.1
2.1 Introduction
With the advent of increased amounts of bandwidth, the communication between
software vendors and their customers can greatly be improved by introducing automatic
error feedback reporting, usage feedback reporting, electronic customer feedback, and
license, patch, and update distribution. Whereas in the past customers and vendors
could only communicate by mail and phone, the World Wide Web can now function
as a lifeline between customers and software vendors, allowing automatic license
retrieval, deployment and error feedback, automatic updates, and automatic provision
of commercial information to customers. Product software vendors, however, generally
do not implement any of these key practices.
To date product software is a packaged configuration of software components or
a software-based service, with auxiliary materials, which is released for and traded in
a specific market [132]. Product software vendors encounter many problems when
1This work was originally published in the proceedings of the 22nd International Conference on Software
Maintenance, entitled “Definition and Validation of the Key Process Areas of Release, Delivery and
Deployment for Product Software Vendors: turning the ugly duckling into a swan” in 2006 [143]. The
work is co-authored with Sjaak Brinkkemper.
23

SECTION 2.2 Process Areas for Customer Configuration Updating
Cust
ome
r
Orga
nizat
ion Sales
Inform
ed
Custo
mer
Unin
forme
d
Custo
mer
Adve
rtise
Upda
te
Rece
iveIn
fo
Custo
mer
Poss
ess
esU
pdate
Rollb
ack/
Dein
stall
Rece
iveU
pdate
Deliv
erUp
date
Insta
lled
Custo
mer
Deplo
y/Ins
tallU
pdate
Deplo
ymen
t
Feed
back
Activ
ated
Custo
mer
Deac
tivate
Activ
ateU
pdate
Rem
ove
Reco
nfigu
re
Conf
igure
Vend
or
Rele
ase
Repo
sitory
Prod
uct R
1
Prod
uct R
2
Prod
uct R
n
. . .
Deplo
ymen
t
Supp
ort Usag
e
Supp
ort
Licen
ses
Usag
eFe
edba
ck
Licen
se(s)
Customer Relationship Management System
SW
Deve
lopm
ent
Softw
are
Conf
igura
tion
Mana
geme
nt
Syste
m
Activation &
Usage
DeploymentDeliveryRelease
Figure 2.1: CCU Model
in Section 2.3. A description of the results per case study is also provided there. The
key practices and combined results of the case studies are discussed in Section 2.4,
where we also defend the claims made in the chapter. Finally, our conclusions and
future work are presented.
2.2 Process Areas for Customer Configuration
Updating
In this section the key process area of customer configuration updating is modelled.
This model explicitly defines customer actions, enabling a software vendor to better
manage and predict the key practices that need extra focus. Much akin to the
CMM [98], the model uses the concepts of key practices, features, and process
areas. Key practices are practices of a software vendor that enable features. Features
are defined as properties of a process that improves product quality and quality
of service. Each process area identifies a cluster of related features that, when
performed collectively, achieve a set of goals considered important for enhancing
process capability. A software vendor possesses a feature within a process area, once
it responds correctly to one of these customer triggered actions.
To describe the key practices for CCU, its process areas need to be established.
These process areas are found using a model for software updaters that focuses
on the customer, which is described in chapter 5. Due to the fact that software
maintenance and deployment focuses solely on the customer, the model is extended
25

Definition and Validation of Customer Configuration Updating CHAPTER 2
with the organizational interactions that are required to fully support a customer’s
actions after an update is released. The CCU model, as seen in Figure 2.1, displays
the states a customer can move through after a product or update release on the
right side. On the left side, the organizational structures that facilitate interaction are
displayed. Within the CCU model four process areas are distinguished, being release,
delivery, deployment, and the activation and usage process areas. The process areas are
separated by dotted lines in figure 2.1 and are further described in the sections below.
Both the process models of the Software Dock [21] and SOFA [99] are contained in
the presented model.
Processes in the model are triggered by customer actions. These actions
are becoming aware of, downloading, deploying, reconfiguring, activating, and
deactivating the release. When a vendor receives a customer request, the Customer
Relationship Management (CRM) system is used to identify the customer. The vendor
then handles the request and interacts with the customer. The customer moves through
a number of states when about to update his configuration. At first the customer is
unaware of the update, until the customer requests information about a product. Once
received, the customer hopefully downloads, deploys and activates it for use, in the
mean time communicating with the vendor in the form of software, licenses, feedback,
and product knowledge.
2.2.1 Release Process Area
The release process area describes the release of a software product for a specific
vendor and the interaction with its customers. The features within the release process
area are:
 Release process management
 Product knowledge management
With respect to release process management a primary key practice is a formalized
release procedure describing step by step how a release is created. Another key
practice is the sharing of knowledge within the organization about the next release,
such that all employees whose jobs are in some way related to the new product
release are aware of the functionality in the next release, the release date, and the
policy on sharing such information. Such awareness creates transparency within
the software development organization, improving the relationship between the sales
and development departments. This is related to the key practice that the sales,
development, and support departments must all be aware of the product’s relationships
with other components, such that no late surprises at a customer site are possible. For
example, if a product comes in simplified Chinese, it might not be compatible with a
large number of commercial database management systems, even though the original
release of the application in English did work.
One key practice of the release process area with respect to product knowledge
management is that all versions of the software that have been released by an
organization, must be stored in a release repository that mirrors the releases in the
software configuration management system. This enables customers using older
26

SECTION 2.2 Process Areas for Customer Configuration Updating
versions to reinstall and update their product at any time. The same way releases must
be managed explicitly, the software vendor must manage explicitly all internally used
development and CCU support tools. Finally, the vendor must manage all external
components that are included and packaged with the product.
The vendor must make a conscious effort to keep its customers updated on the latest
news and product releases using any channel of communication, such that customers
are not lagging behind in either product releases or product release information. Sales
and lead management includes the use of pilot customers that pre-evaluate and test
the software before an official release. Also, customer communication in the release
process area is most interesting to the sales department of a product software vendor.
A sales department must have insight into the purchasing guidelines and processes of a
customer organization. One relevant aspect to determine product quality is strategic
planning of product releases and updates. Customer organizations utilize product
software in such an intensive manner that an update is a costly matter, due to down
time, system instability, and the number of systems that require the update. A software
vendor must establish the best time when an update is published and what the possible
consequences are of deploying the update [5]. Microsoft, for instance, releases its
security updates for all its products on the second Tuesday of the month and they have
communicated this with their customer base.
2.2.2 Delivery Process Area
The delivery process area concerns the delivery of software, licenses, and product
knowledge to customers. The key practices belonging to the delivery process area
are focused on the following features:
 Delivery methods to customers
 Customer side delivery
To begin with software vendors must enable customer organizations to perform
deployment using whichever medium a customer chooses, such as DVD, CD, a local
area network, or the Internet. Secondly, customers must be able to remotely deploy
applications and updates onto a user system without physically having to touch it.
Thirdly, the product must supply a mechanism for automatic pull of updates, such
that the customer can check for updates and download them automatically on a regular
basis. The customer must be able to abstract from the download site of the vendor,
allowing the customer to use an internal download server. If possible, the product must
send back a deployment report after a customer has deployed the product, to inform
the vendor whether the deployment was successful or not.
2.2.3 Deployment Process Area
The deployment process area contains key practices that enable a product to be
deployed, removed, and updated. The key practices in the deployment process area
are categorized into:
 Environment checking
27

Definition and Validation of Customer Configuration Updating CHAPTER 2
 Local configuration management
 Deployment process automation
The key practices related to local configuration management are prone to many
issues, such as missing (external) components, incomplete downloads, erroneous
deployments, and overwritten customizations. To improve the deployment a
deployment tool must inspect the local configuration, to see whether external
components are missing and whether the local system provides enough resources,
such as disk space. Also, downloaded packages must be checked for integrity and
completeness. In the cases of missing components and files that do not pass their
integrity checks, some automatic resolution must be implemented. Finally, it must
be possible to rollback from an update or deployment to return to the previous
configuration.
Customizations are widely applied for specific business domains and for specific
customers. In many cases these customizations account for a large portion of
their total revenue, which proves that explicit customization management is vital to
many software vendors. A key practice for a software product with many different
customizations at different customers is that the main product is updated without
overwriting local customizations.
Once these issues have been tackled [138], the software vendor can make these
processes as quick and easy for the customer as possible by implementing semi-
automatic deployment, update, and rollback procedures. Another key practice is that
updates do not require downtime when performing an update, allowing the customer to
use the product without interruptions.
Table 2.1: Some Statistics on each organization
Software Employees CCU Customers Technology CMS
Vendor employees
ERPComp 1500 15 160.000 ASP+ Delphi Proprietary
CMSComp 65 5 140 Java SubVersion
FMSComp 160 3 900 Delphi + Java VSS + CVS
OCSComp 115 2 20 C++ CVS
CADComp 60 3 4.000 Delphi PVCS
HISComp 100 2 40 Delphi VSS
Mozilla 710 5 to 10 1.000.000+ Java CVS
Apache 388 NA 1.000.000+ Java SubVersion
Customer organizations often use different testing and acceptance stages according
to the IT Infrastructure Library (ITIL) [22] before actually implementing software in
the entire organization. This requires that deployments are done quickly, and that
configuration settings and data files are moved separately from the software. This
key practice is related to the externalization of all user and configuration data, which
enables a transparent configuration environment [36]. Within such an environment all
configuration and user data is accessed externally from the product, which allows for
28

SECTION 2.3 The Cases and their Key Practices
relationships to be established between configuration data between products, enabling
sharing of user configuration data such as e-mail account settings between e-mail
clients, font sizes between applications, or even appearance settings between operating
systems. Such externalization allows for the product to perform product data backups
as well, enabling quicker and more reliable backup retrieval actions.
2.2.4 Activation and Usage Process Area
The activation and usage process area concerns the activation and working of a
product at the customer site. The activation and usage process area focuses on the
following features:
 License management
 Feedback management
License management enables a customer organization to manage licenses
explicitly, and activate the product with a different license on each start-up, allowing
customers to use test and development versions, and to provide different functionality
to different user profiles. Another key practice belonging to license management is that
licenses need to be stored in some coded fashion, to hinder piracy of products. Finally,
to have maximum commercial flexibility, the licenses should control large parts of the
software, such that any functionality is activated or deactivated using the licenses.
The vendor must also explicitly manage its customer licenses. To begin with, a
vendor must be able to automatically renew a license for a customer, such that the
vendor can renew or prolong a license without much effort. To achieve this, it must be
possible to generate licenses from contracts automatically.
Feedback management allows a vendor to gather large amounts of data about
its customers and its product as it acts in the field. Feedback can come from
either automatic sources or manual customer triggered sources. Feedback is used,
in the automatic case, to provide knowledge to the vendor about product usage and
knowledge about the customer’s configuration. Finally, the user should be able to report
errors and questions to the software vendor through the software product. This allows
users to state questions and report bugs about specific screens and unclear functions in
the product.
2.3 The Cases and their Key Practices
In this section the anonymized cases are described. Some generic information is
provided on each software vendor and the reasons why the case was included in this
research are stated. A description is also given on how the case studies were performed.
Table 2.1 provides some statistics on each organization that is part of our research
set. Tables 2.2, 2.3, 2.4, and 2.5 show the key practices these software vendors have
implemented. Each of the key practices has been evaluated using a list of criteria,
which have been left out for the sake of brevity. An open circle shows that the vendor
has implemented the facilities that provide a key practice, yet it has not implemented
the key practice. These can be considered “quick wins” for the vendor.
29

Definition and Validation of Customer Configuration Updating CHAPTER 2
2.3.1 Case Study Approach
To produce these results six descriptive case studies [133] were performed at Dutch
software vendors. These case studies resulted into six case study reports [19]. During
several months of doing the case studies, facts have been collected from several
sources:
 Interviews - To study the cases and confirm our hypotheses, interviews were
held with the people responsible for the development and usage of the studied
product.
 Studying the software - Academic licenses were granted to the products. These
licenses helped to gather many facts by examining the products, using the
products, and experimenting with the product and its updating capabilities.
 Document study - Document study was performed to evaluate the development
process and cross check the answers provided by the other sources of
information.
 Direct observations - Since our research took place at the development
departments (of the non-open source cases), we were able to directly observe
and document day to day operations.
The interviews consisted of two sessions, one to explore and elaborate, and one
to cross-reference answers from other interviews. The second session was also used
to cross-reference documentation and confirm the facts stated in these documents.
Besides these reviews we also created a case study protocol and a case study database.
To ensure reliability, the case study report was reviewed by key informants. Two open
source organizations were included to evaluate their key CCU practices. For these
two cases all on-line material was used, including the source code of the products
and the products themselves were tested extensively. The open source cases’ high
numbers of employees can be explained by the fact that open source developers are
not working on a product full-time. The open source cases can therefore not be
compared to the commercial cases in terms of size. The open source cases have been
added to show that the CCU model can be used for any type of software vendor or
distribution organization. Also, the open source cases contrast with the commercial
cases in a number of interesting ways. CCU model coverage looks different for an
open source product than for the other products presented. To begin with, licensing is
an underrepresented aspect of open source products for obvious reasons and bugs tend
to be reported using other channels than the product itself (Bugzilla, for instance).
The validity threats to our case studies are construct, internal, external, and
reliability [133] threats. With respect to construct validity, the same protocol was
applied to each case study, which was guarded by closely peer reviewing the case
study process and database. To create a complete and correct overview, both the
development and CCU processes have been documented extensively. The internal
validity was threatened by incorrect facts and results from the different sources of
information. By crosschecking these results and observing the processes as they were
going on a complete view could be created. With respect to external validity, the cases
are representative for the Dutch software vendor market domain because each software
vendor has a different number of customers and is active in a different problem domain.
30

SECTION 2.4 Evaluation of the Process Areas and Features
dependency resolution. Also, the availability of past product releases is required
such that customers can download older versions. Mozilla does not provide such
functionality, due to the fact that the source of their products is always available. The
downside of this is that a customer can never download older versions of the software
automatically to be used with a set of other applications, without having to build the
source code. Another example is ERPComp, which only provides the latest version
of the software and no other, such that users will always use the latest version. The
question remains whether a vendor wishes to provide customers with more flexibility,
or whether this simplification and therefore cost saving method does not scare off
customers. Many tools are available for knowledge management and distribution,
however, each organization has its own channels for distribution.
The delivery methods to customers feature is dependent on many different factors,
such as bandwidth, network policies, security, and infrastructure. Coverage of all key
practices in this process area is rather weak, with the automatic pull key practice as an
extreme. To improve in this process area, a software vendor must carefully review
whether the software architecture and the vendor organization itself do not restrict
customer communication. Integration of the CRM system throughout the complete
organization is required to gain serious improvements in this area. There are some tools
available that are already supplying such integration, though a lot of customization is
required [148].
The customer side distribution feature is dependent on the format of deployment
and installation packages, the product software architecture, and possible storage
locations. The key practice to allow a customer to use any medium for deployment
enables customer organizations to freely deploy software using its proprietary methods
of deployment. An example encountered in the cases is when customers request for
Microsoft Installer packages (msi) because their internal deployment and distribution
tools require msi packages. Since the customer does not allow each user system to go
on-line individually and download the latest updates from the vendor but forces them
to download patches and products locally, much bandwith is saved. Tool support is
found in package managers, such as rpm-update, Portage, and Microsoft’s open source
project Wix [130].
Environment checking, a feature of deployment, requires the deployment
application or software product to first scan the system on which the update will take
place, for the availability of required components and possible resource constraints
such as disk space. If such constraints or missing components are encountered, these
issues must be resolved automatically. There are not many tools available (besides
package managers) that can support these key practices, mostly due to the complexity
and number of different deployment environments.
The Local configuration management feature is highly dependent on the
operating system and deployment tools used by the customer. Some deployment tools
have integrated the build process into the management of software packages [33],
whereas other tools are primarily focused on copying of files from one location to
another, such as InstallShield [58]. Implementing the key practices in this process
area requires large development efforts and changes to the software architecture, such
as the rollback key practice, which is generally not implemented because changes
to the data model cannot be rolled back [64] [62] without a versioned database
35

SECTION 2.5 Discussion
some of its primary customers. New functionality can also be made available to
customers using temporary licenses, which allows customers to test new functionality
before actually purchasing it. This process area is limited by trust and network
infrastructure issues. It will require some changes to the CRM system to get messages
to the right customer organizations. Some commercial tools are available in the form of
PDM and CRM systems, but once again integration and customization effort are large,
and structural communication between the sales and development departments about
new product features is required.
The feedback management feature is valuable to a software vendor because it
will introduce new requirements on the product, show what the most used functions of
a product are, and where most errors occur. Though improvements in this process area
requires a lot of effort, the products discussed in this chapter already implemented
different error logging mechanisms, sometimes even with a “send error report to
vendor by e-mail”-button. No commercial tools were found to handle such feedback
although some tools such as Mozilla’s TraceBack and the components presented in the
work by Renaud et al. [102] provide similar functionality. Network infrastructure,
privacy, and security should be taken in consideration carefully when improving
these areas. Effort to improve this feature is low, whereas the implementation of
feedback is highly profitable. Such feedback reports can even be linked to customers,
informing the vendor of its customers’ configuration. This information is used by
the support department to determine a customer’s configuration, but also to inform
the development department of “proven” configurations. A well-known example
of feedback error reporting is the feedback function in Microsoft’s Windows XP.
However, other mechanisms are imaginable [41], such as usage reports (which can
also be used for pay-per-usage scenarios) that can help improve the knowledge about
which functionalities are most used by customers.
2.5 Discussion
Now we put the key process area of CCU up for discussion. The first question that
needs to be answered is whether a software vendor’s success relies on its customer
relationships. In the commercial cases encountered and presented in this chapter 50%-
70% of their yearly revenue was coming from existing customers, which in our view
shows that customer retention and the maintenance of relationships is essential to
survive in the current industry. In the case of open source products, where many
of the users of the product are also developers, testers, and quality assurance team
members, the same premise on customer relationship management holds. Since CCU
is a customer focused process, the improvement of these processes will lead to better
customer relationships and possibly a higher customer retention rate. By applying the
CCU model onto the eight presented cases, Tables 2.2, 2.3, 2.4, and 2.5 lead to the
following observations:
 Software vendors focus insufficiently on customer side configuration
management
 Licensing and contract integration is rare
37

SECTION 2.6 Future Work
In our search for tools that can provide the key practices presented in this chapter
and in chapter 5, undiscovered product niches have been encountered. To begin with
it seems that there are no product data management systems that explicitly manage
licenses, software products, their fixes, and their patches, in such a way that customers
can log in and download them. Secondly, feedback sending and feedback analysis
applications seem to be in short supply. Finally, operating systems and deployment
tools [21] generally do not support the key practices for local software configuration
management.
If anything can be learned from this research, it is that software vendors must
integrate their CRM, PDM, and SCM [148] systems to automate the processes related
to CCU. Such automation provides more efficient methods to perform repetitive tasks
such as license creation, license renewal, product updating, error reporting, usage
reporting, product release, and manual configuration tasks, such as backups. The
second main lesson is that usage of feedback reports supplies software vendors with
the largest test bed imaginable, and therefore deserves more attention. The presented
CCU model can be used as a guideline for software vendors or for the development of
a software manufacturing and software product data management system.
2.6 Future Work
The presented material allows for a larger evaluation of the customer configuration
updating process. Next to the case studies we will perform in the future, we are
planning to build a benchmark site where software vendors can evaluate their own key
practices and position themselves in the market. This evaluation technique, however,
requires a new classification of software product companies, which is used to further
analyze the results from such research. As a continuation on one of the cases we
have been offered the opportunity to implement a subset of the presented key practices
within that organization. We will investigate the implementation of these key practices
and use it to validate the results of this research. We are currently negotiating with
several software vendors whether the concepts shown by Elbaum et al. [41] can be
implemented within their products, to evaluate the usefulness of such functionalities
and to get more practical experience with field data gathered from customers.
39

Definition and Validation of Customer Configuration Updating CHAPTER 2
D
ep
lo
ym
en
t
K
ey
P
ra
ct
ic
e
So
ft
w
ar
e
pr
od
uc
t
ve
nd
or
s
E
R
P
C
C
M
S
C
F
M
S
C
H
IS
C
C
A
D
C
O
C
S
C
M
oz
il
la
A
pa
ch
e
C
on
fi
gu
ra
ti
on
co
m
pl
et
en
es
s
ch
ec
ki
ng
of
ex
te
rn
al
co
m
po
ne
nt
s







A
ut
om
at
ic
re
so
lu
ti
on
of
de
pe
nd
en
cy
is
su
es

(S
em
i-
)a
ut
om
at
ic
lo
ca
lu
pd
at
e
pr
oc
es
s





R
ol
lb
ac
k
fr
om
an
up
da
te
is
po
ss
ib
le


R
ol
lb
ac
k
fr
om
an
in
st
al
li
s
po
ss
ib
le


U
pd
at
es
re
qu
ir
e
no
do
w
nt
im
e



Te
st
,a
cc
ep
ta
nc
e,
pr
od
uc
ti
on
en
vi
ro
nm
en
ts





U
pd
at
es
ca
n
co
pe
w
it
h
lo
ca
lc
us
to
m
iz
at
io
ns





-

E
xt
er
na
lc
on
fi
gu
ra
ti
on
to
al
lo
w
tr
ac
e




In
te
gr
it
y
ch
ec
ks
of
S
W
ar
te
fa
ct
s
at
cu
st
om
er


-


D
et
ec
ti
on
an
d
ex
pl
or
at
io
n
of
cu
st
om
er
en
vi
ro
nm
en
t

-


D
at
a
ba
ck
up
s
ar
e
do
ne
th
ro
ug
h
th
e
pr
od
uc
t





L
eg
en
d:
:
C
ur
re
nt
ly
im
pl
em
en
te
d;
-:
N
ot
ap
pl
ic
ab
le
;e
m
pt
y:
N
ot
im
pl
em
en
te
d

R
eq
ui
re
s
so
m
e
m
an
ua
ls
te
ps
,y
et
w
ou
ld
be
ea
sy
to
au
to
m
at
e;
Table 2.4: Deployment key practices
42

SECTION 2.6 Future Work
A
ct
iv
at
io
n
an
d
U
sa
ge
K
ey
P
ra
ct
ic
e
So
ft
w
ar
e
pr
od
uc
t
ve
nd
or
s
E
R
P
C
C
M
S
C
F
M
S
C
H
IS
C
C
A
D
C
O
C
S
C
M
oz
il
la
A
pa
ch
e
L
ic
en
se
s
ar
e
co
de
d





L
ic
en
se
s
ha
ve
an
ef
fe
ct
on
so
ft
w
ar
e






L
ic
en
se
s
ar
e
m
an
ag
ed
ex
pl
ic
it
ly
by
cu
st
om
er



P
os
si
bl
e
to
re
ne
w
li
ce
ns
es
au
to
m
at
ic
al
ly




L
ic
en
se
s
ar
e
ge
ne
ra
te
d
us
in
g
co
nt
ra
ct
s



Te
m
po
ra
ry
li
ce
ns
es
ar
e
di
st
ri
bu
te
d

A
w
ar
en
es
s
of
cu
st
om
er
s
co
nfi
gu
ra
ti
on






F
ee
db
ac
k
fr
om
a
us
er
is
po
ss
ib
le
th
ro
ug
h
th
e
pr
od
uc
t


U
sa
ge
re
po
rt
s
ar
e
cr
ea
te
d




L
eg
en
d:
:
C
ur
re
nt
ly
im
pl
em
en
te
d;
-:
N
ot
ap
pl
ic
ab
le
;e
m
pt
y:
N
ot
im
pl
em
en
te
d

R
eq
ui
re
s
so
m
e
m
an
ua
ls
te
ps
,y
et
w
ou
ld
be
ea
sy
to
au
to
m
at
e;
Table 2.5: Activation and usage key practices
43

C H A P T E R 3
A Case Study in Mass
Market ERP Software
The maintenance of enterprise application software at a customer site
is a complex task for software vendors. This complexity results in a
significant amount of work and risk. This article presents a case study of a
product software vendor that tries to reduce this complexity by integrating
Product Data Management (PDM), Software Configuration Management
(SCM), and Customer Relationship Management (CRM) into one system.
The case study shows that by combining these management areas in a
single software knowledge base, software maintenance processes can be
automated and improved, thereby enabling a software vendor of enterprise
resource planning software to serve a large number of customers with
many different product configurations.1
3.1 Integrated Development and Maintenance
The complexity of the maintenance, release, and deployment processes of product
software is a result of the enormous scale of the undertaking. There are many
customers for the vendor to serve, who all might require their own version or variant
of the application. Furthermore, the application itself will consist of many (software)
components that depend on each other to function correctly. On top of that, these
components evolve over time to answer the changing needs of customers. As a
consequence, the release and deployment of these applications take a significant
amount of effort and is a time consuming and error-prone process.
To date product software is a packaged configuration of software components or
a software-based service, with auxiliary materials, which is released for and traded
in a specific market [132]. Customer Configuration Updating (CCU) is defined as
the combination of the vendor side release process, the product or update delivery
1This work was originally published in the Journal of Software Maintenance and Evolution: Research
and Practice, entitled “Integrated development and maintenance for the release, delivery, deployment, and
customisation of product software: A case study in mass market ERP software” in 2006 [141]. The work is
co-authored with Sjaak Brinkkemper, Gerco Ballintijn, and Arco van Nieuwland.
45

A Case Study in Mass Market ERP Software CHAPTER 3
process, the customer side deployment process, and the activation process [143]. To
alleviate these processes I envision a Software Knowledge Base (SKB) that contains
facts about all product artefacts together with their relevant attributes, relations and
constraints [19]. In this way, high-quality software configurations can be calculated
automatically from a small set of key parameters. It also becomes possible to pose
“what-if” questions about necessary or future upgrades of a customer’s configuration
[142]. The need for a distributed software knowledge base comes from literature.
Meyer is the first one to introduce the concept of a centrally available software
knowledge base [80]. Others, such as Klint et al. [70] and Robillard et al. [104]
emphasize the need for explicit knowledge management during development and
maintenance.
Exact Software (ES)2, a Dutch software manufacturer that serves 160,000
customers worldwide, has implemented an SKB to manage and improve its software
maintenance, release, and deployment processes. The SKB used by ES is
implemented in their own commercial product called e-Synergy. In this article I
show that ES successfully supports its large customer base with an integrated product
data management, software configuration management, and customer relationship
management system, thereby alleviating the process of software product maintenance.
The article describes how the processes of development, release, and deployment have
been improved by integrating processes that were previously managed by utilizing
different isolated systems. The article also demonstrates how a central software
knowledge base, containing all the relevant knowledge about software products, is
implemented and used to support the processes of software maintenance. Finally, the
article describes four principles employed by ES to deal with general complexities in
the software engineering discipline with respect to software maintenance.
Figure 3.1 displays the overall architecture of the integrated CRM, PDM, and SCM
systems in e-Synergy. The integration of these systems enables efficient maintenance
of software configurations at a customer site. The CRM system contains a contract
module, in which all products that have been sold to a customer are stored. Each
contract applies to a product that can be downloaded and activated by a customer. The
products stored in the PDM system are associated with their corresponding artefacts
in the SCM system, which enables a customer to download the correct files that are
required for a product update or deployment. Finally, the PDM system generates a list
of files on the vendor side that is compared to the list of files on the customer system.
When differences are encountered the required files are downloaded by the customer
automatically to update the customer configuration.
The software maintenance processes on the vendor side also have improved by
the integration of different systems. The integration of the SCM and PDM systems
allow product managers to quickly oversee whether work still needs to be done on
deliverables before the next release. The integration of the SCM and workflow
management system enables traceability of changes on deliverables, thereby improving
product quality. The integration of the SCM and PDM systems also allows for quick
deployment and testing of test versions for the quality assurance department. These
2Please note that this research took place in 2003 and 2004. Some of the presented results no longer
represent daily ES practice
46

SECTION 3.2 Research Approach
CRM
Custo
mer
Data
base
Custo
mer

1
Custo
mer

2
Custo
mer

3
Custo
mer

4
Custo
mer

5
Cont
racts
Data
base
Cont
ract
3,1
Cont
ract
2,2
Cont
ract
2,1
Cont
ract
1,2
Cont
ract
1,1
PDM
Prod
ucts
Prod
uct 1
Prod
uct 2
Prod
uct 3
Prod
uct 4
Prod
uct 5
Prod
uct L
ines
Prod
uct Line
1
SCM
Deliv
erabl
e
Artifa
cts
Bina
ry 1
Bina
ry 2
Bina
ry 3
Bina
ry 4
Bina
ry 5
Sour
ce fil
es
. . .
Sour
ce 3,
1
Sour
ce 2,
2
Sour
ce 2,
1
Sour
ce 1,
2
Sour
ce 1,
1
Prod
uct Line
3
Prod
uct Line
2
Bina
ry 6
Bina
ry 7
Bina
ry 8
. . .
. . .
. . .
. . .
. . .
Figure 3.1: Integration of CRM, PDM and SCM
and other improvements are discussed further in Section 3.3.
The rest of this article is structured as follows. Section 3.2 describes the objective
of our research at ES and a motivation. Section 3.3 describes ES and the tools it uses
to integrate its SCM, PDM, and CRM systems. Section 3.4 describes the maintenance
processes of the products at the vendor site and of the configurations at the customer
site. Section 3.5 discusses the lessons learned from ES and what functionality I feel is
lacking in Es’s SKB. Related work is presented in Section 3.6 and finally Section 3.7
concludes our article with a discussion.
3.2 Research Approach
3.2.1 Problem Overview
Two important parts of the maintenance process are release and deployment of
software. The release and deployment processes for a software product involve a large
amount of risk and effort for a software vendor. These processes, however, have been
documented insufficiently in literature. The SWEBOK [1], for instance, gives a generic
description in the SCM chapter of the processes of release and delivery. The Capability
Maturity Model (CMM) [98] also does not provide adequate descriptions for these
processes, which is explained by the fact that the CMM does not focus on product
software specifically. Attempts have been made in the release candidate of the IT
Service CMM [86], although the IT Service CMM also does not provide an elaborate
description for the processes of release, delivery, and deployment. Clearly, even though
47

A Case Study in Mass Market ERP Software CHAPTER 3
there is a need for process definitions, there are no adequate process descriptions
available. The goal of our research is to simplify the software release and deployment
effort. I propose to do so by managing all the knowledge about a software product
explicitly. The explicit management of software knowledge enables the evaluation of
“what if” scenarios, such as, what will happen to the current configuration of customer
X, if she upgrades application component Y? These evaluations help in assessing the
risk of the deployment process, and these assessments, in turn, improve interaction
between customer and software vendor because the vendor can guarantee whether a
combination of components can function correctly together.
Managing software knowledge is, however, only part of the story. The software still
has to be delivered to customers. I aim to support dynamic delivery of software via the
Internet, both in the form of upgrades and of full packages. The previously mentioned
product and component knowledge is used to compute the difference between the
existing software configuration at a customer and the desired configuration. This
difference is used to create required upgrades [124]. Central to the maintenance
activities I envision, is the software knowledge base. This software knowledge base can
be seen as an integrated SCM/PDM/CRM system that stores all information about all
the artefacts that are part of the applications life cycle. The SKB stores the information
of all available applications in all available versions at the vendor site, whereas at the
customer site the SKB stores information about the installed applications, application
settings, and configurations. Both the vendor and the customer can request or receive
information from the configuration of the other party, such as regular updates, product
information, usage and error reports, product knowledge, and licenses.
As part of our research, I have been conducting case studies [62] [7] at product
software companies to evaluate the state-of-the-practice of software vendors in the
Netherlands, such as ES. ES is relevant to our research because ES has implemented
one of its own products, e-Synergy, to support the processes of release and deployment
and to function as an SKB, which partly validates the theory that an SKB can improve
software release and delivery.
3.2.2 Exact Software
ES is a manufacturer of software for accounting and enterprise resource planning
(ERP), based in Delft, the Netherlands. Since its founding in 1984, ES has established
an international customer base of over 160,000 customers, mainly in the small to
medium enterprise sector. Through autonomous growth and a number of acquisitions
the number of employees has grown to 2025 in 2004 (see Table 3.2.2). Twenty percent
of these employees are active in the development of software on several international
locations with most of them (180 employees) working in Kuala Lumpur.
A typical application sold by ES is Exact Globe, a back-office application that
integrates business processes, such as finances, logistics, PDM, and CRM. A recent
product is e-Synergy, a front office application that provides organizations with
real-time financial information, multi-site reporting, and relationship and knowledge
management capabilities. Employees, customers and partners are provided with real-
time access to information across an entire organization.
Based on more than 20 years of experience in developing software products for
48

SECTION 3.2 Research Approach
Department FTE Percentage
Support 546 27,0
Services 263 13,0
Sales and Marketing 445 22,0
Finance and Administration 142 7,0
Staff and General Management 223 11,0
Development 294 14,5
Quality Assurance 96 4,7
Release and Deployment 15 0,7
Total 2024 100
Table 3.1: ES full-time employment (2004)
the small to medium enterprise market, ES enforces four main principles for product
development:
 Uniform architecture - All software developed by ES has a three layered
architecture. The user application layer (a browser or a standalone client), the
application server layer (containing the business logic), and the database layer.
 One-X - ES has developed a strategy for developing its ERP software, called
One-X, which aims to develop all software around one single instance of truth,
making the data available to all ES applications, such that the data can be created
and provided to all the stakeholders. The idea behind One-X is that data needs to
be entered just once and that extensive navigation is possible through integration.
 KISS - To support such a large customer base within such a complex problem
domain, ES follows the principle of KISS (Keep It Small and Simple) for its
development process. The use of KISS within ES has resulted in a development
cycle where a fully functional prototype is produced by a spearhead team first.
Once the prototype is released, the product enters a maintenance cycle and
the product can then only be changed by the maintenance team through well-
defined maintenance procedures. All procedures are monitored by a large quality
assurance team, as seen in Table 3.2.2. These procedures allow ES to keep the
maintenance of its products simple and controlled.
 Eat your own dogfood - ES uses its own software products to support internal
processes, which is called “eat your own dogfood” by Microsoft [28]. This
internal use provides the maintenance department with early bug reports and
feedback.
These principles are enforced with one fact in mind. Revenue statistics from ES
show that since 2003, the largest stream of income cines from maintenance contracts,
whereas previously money came for the larger part from license sales. This made ES
management realize that investment in the maintenance process was required.
49

A Case Study in Mass Market ERP Software CHAPTER 3
Deliv
erabl
e1 Sour
ceFi
le 1
Deliv
erabl
e2 Sour
ceFi
le 2
Sour
ceFi
le 3
One-
of
More
-of
Man
dato
ryCom
pany
Licen
se
Unive
rsity
Licen
se
Com
pone
nt 1
Com
pone
nt 2
Com
pone
nt 3
Manu
al
Box Deliv
erabl
es
Sales
View
Deve
loper
View
Sale
sIte
m Pr
oduc
t X
Figure 3.4: Sales and Developer Product View of an Example Product
ES’s products are represented in e-Synergy in different ways at the development
sites of ES, whereas instantiation information for products at customers is stored in
the contract management facility, the product data management facility, and locally at
the customer’s site. These two different views are based on the assumption that sales
personnel do not need to know all the implementational details, whereas developers
are not concerned with sales knowledge. An example is that the sales department sees
products as decomposable modules that can be sold separately, whereas development
sees the product as a large set of deliverables containing all modules, which are later
activated or deactivated at runtime by the license file. Figure 3.4 displays a generic
product structure and the two different views of that structure:
Sales View - From the point of view of the sales department it is not relevant
what deliverables and sources look like. For this view it is required to know what a
product costs, what options there are to a product, what kind of sales agreements are
possible, and what materials make up a product. Sales personnel thus share no interest
in source files. A product, consisting of sales items connected by the one-of, more-of,
mandatory, and optional relationships, is instantiated by binding these relationships.
The binding of these relationships corresponds with the product information stored in
a license file. The relationships defined here are similar to the relationships defined for
feature diagrams [67, 123].
Development View - Developers are concerned with deliverable files and source
files. As developers always work in the context of a product, and they know the
complete structure of that product, however, developers usually do not use sales items.
A developer considers a product as a complete set of deliverables, therefore there are
54

A Case Study in Mass Market ERP Software CHAPTER 3
3.4 Maintenance and the SKB
3.4.1 Maintenance at the Development Site
Before the systems of SCM, PDM, and CRM were integrated, the concurrent
versioning system RCS [110] was used to support development. Besides RCS, ES
used daily build servers, so that the quality assurance department could check the
work of the day before. During the implementation of e-Synergy, ES drew up standard
requirements for change control, team support, status reporting, process control, and
audit control. Aside from these standard requirements, ES had the following non-
generic requirements [76] in the area of SCM:
 Version control - In the past too many resources were absorbed by legacy
support and customers were confronted with complex upgrades and bug fixes.
As a consequence of the KISS principle, ES decided to reduce complexity for the
customer and development by no longer supporting and storing multiple versions
of a product.
 Configuration support - Because bandwidth and disk space are relatively
cheap and development is expensive, ES concluded that installing the full set
of deliverables for a product and activating purchased modules according to a
license is more efficient than doing partial delivery. Also, to improve service
and product quality, ES wished an automated check of the validity of a product
configuration before it is deployed.
 Build support - ES previously used separate build servers to build products
overnight. That build was tested the next day by quality assurance. As ES grew
larger internationally and developers were working on code twenty-four hours a
day, there was no down time left for servers to build the software. A new way of
partially building was required to facilitate these needs.
ES promotes some key starting points for its development process. To begin
with, developers have private ownership of deliverables and source code and they
include their compiled deliverables when comitting their source code. This introduces
pessimistic locking (where no two developers can be working on one source file)
and enables management to assign responsibility for deliverables to one developer
specifically. ES uses very strict maintenance procedures and role based workflow for
each task such as functional requests or bugs. First the development manager evaluates
the task. Once evaluated the developer executes the task. Finally, quality assurance
tests whether the task was successful and approves the task.
ES has development sites in multiple time zones covering twenty-four hours, which
eliminates the option to perform nightly builds. This caused the decision for building
the end product on the developer’s workspace. The solution where developers upload
their compiled deliverables creates a repository that always contains the most recent
build of the software, removing the need for nightly builds. This solution also enables
ES to have a latest running version available twenty-four hours a day at all departments
worldwide.
 Manage deliverables - Previously, source files were the focus of management
instead of deliverables. A compiled version of the software created on the build
56

SECTION 3.4 Maintenance and the SKB
directly to a customer, whereas in the past its licensing was done through license
numbers provided with the distribution CD-ROM.
Custom Solutions
Ever since ES produced standard out-of-the-box applications a Custom Solutions
department has been needed to create specific solutions for customers. To do so in
e-Synergy a specific messaging architecture has been created to enable the addition of
extra components. Custom solutions are created using a dedicated e-Synergy software
development kit (SDK). Custom Solutions produces two types of customisations,
being building blocks and customer specific solutions. Building blocks are standard
customisations that are applicable to specific market niches, such as equipment rental
companies or educational organizations. Customer specific solutions are requested
by customers and are not generalizable to other customers. Finally, customers can
purchase the option to create customisations themselves.
All three types of customisations are built using the same SDK and are facilitated
by a messaging architecture. The messaging architecture created by ES is kept as stable
as possible, such that custom solutions that worked with older versions of e-Synergy do
not break down due to an update. If a customisation changes a file that belongs to the
product itself, such that an update would remove the customisation, the customisation
developer can mark the file as unfit for update. The Product Updater will then simply
skip the file during the update process. Obviously, it is impossible to guarantee that the
product remains fully functional after an update when changes have been made to the
product itself and therefore ES focuses on communicating regularly with its customers
that implement customisations.
When the Custom Solutions department creates a customisation for a customer,
that customisation is stored in a Custom Solutions SCM. The dedicated SCM system
enables customers to update a custom solution automatically when the Product Updater
is run. When a product for which a customisation is built is updated, Custom Solutions
cannot cost effectively test the customisation that was made for one specific customer
in each possible updated configuration and only building blocks are tested for each new
(maintenance) release. However, due to the fact that updates do not break compatibility
in the SDK, customer specific customisations generally do not break down after an
update.
Contracts and License Files
Some of the results of the integration of the PDM and CRM systems are the contract
management functionalities and the license files. The version number that is stored in
the contract management facility for each customer’s product is changed automatically,
when a customer downloads an update, or manually, when a CD-ROM is sent out
to a customer. The version number is used for support purposes, telling the support
department what version a customer is currently using. The link to the customer
support calls and service status is used to see how many calls a customer has made,
and whether a customer is still allowed support. Using the support information leads
to a stricter way of dealing with support and results in less support calls.
59

A Case Study in Mass Market ERP Software CHAPTER 3
3.6 Related Work
The techniques applied by ES to integrate SCM and PDM are similar to those described
by the book on the integration of PDM and SCM systems in Finnish industry [60]. The
execution of case studies performed by Tiihonen et al. [111], Bosch et al. [17], and
Davenport et al. [31] are similar to the way in which the ES case study was performed.
All three describe one or more case studies with qualitative, rather than quantitative
results for software products and development processes. The work presented by
Tiihonen et al. and Bosch et al. is different from our research because it focuses on
the state of the practice of software product configuration in software product lines and
because both apply a problem focused approach. Davenport et al. describe a model of
knowledge management that is later put into practice at Siemens, and differs from our
work in that their model is first compared to the current practices and then implemented
to provide extra grounds for evaluation.
The aim of the presented research is to provide qualitative results to the current
body of knowledge of knowledge management and case studies in product software
companies in software maintenance in general. The techniques applied by ES are
clearly centered around the SCM system, and have been integrated upwards with the
CRM and PDM systems. With respect to the SCM system, I have positioned it in the
framework created by Conradi and Westfechtel [25]. The SCM system under study, by
their definition, is a state based tool-kit that applies a database to store coarse grained
extensional product compositions for any problem domain. The product configurator
applies functional rules to provide an interactional model to its users. Finally, the case
studies performed by us [62] and Ballintijn [7] are of the same format as the work
presented here. These two case studies show, similar to the ES case but to a lesser
extent, that explicit software knowledge management can improve the CCU processes.
The main differences between the ES case and these cases is that they describe much
smaller organizations (160 and 100 employees, respectively) and that they have not yet
integrated PDM, SCM, and CRM to such an extent as ES.
3.7 Conclusion
If anything can be learned from this research, it is that software vendors must integrate
their CRM, PDM, and SCM systems to automate the processes related to CCU. Such
automation provides more efficient methods to perform repetitive tasks such as license
creation, license renewal, product updating, error reporting, usage reporting, product
release, and manual configuration tasks, such as backups. The second main lesson
is that usage of feedback reports supplies software vendors with the largest test bed
imaginable, and therefore deserves more attention. The presented research can be used
as a guideline for software vendors or for the development of a software manufacturing
and software product data management system. In our search for tools that can
provide the key practices presented in this chapter, an undiscovered product niche
was encountered. It seems that there are no (other than ES’s e-Synergy) commercial
product data management systems that explicitly manage licenses, software products,
their fixes, and their patches, in such a way that customers can log in and download
62

A Case Study in Mass Market ERP Software CHAPTER 3
64

C H A P T E R 4
A Benchmark Survey into
the Customer Configuration
Processes and Practices of
Product Software Vendors in
the Netherlands
Product software vendors do not invest enough effort into release, delivery,
deployment, and usage and activation of their software products. Not
spending effort in these customer configuration updating processes leads
to high overhead per customer, which impedes growth in customer
numbers. This chapter presents the results of a survey that provides
product software vendors with an overview of their customer configuration
updating processes and practices, and benchmarks their practices against
competitors using similar technology, of the same size, and active
in the same market. These benchmarks contain customized advice
to the respondent company that can be used to strategically improve
their customer configuration updating processes to gain efficiency and
effectiveness. The survey was held in the Netherlands, and 74 software
vendors responded. Amongst other conclusions, a significant positive
correlation was found between success of a software product and a
vendor’s recent investments into customer configuration updating. 1
4.1 Introduction
The product software industry in the Netherlands is flourishing. Computer games,
ERP products, and navigation systems are just some examples of successful products,
nationally and internationally. Though these businesses have a large body of knowledge
available to them about generic software development and engineering, none of it is
1The work has recently been submitted [146].
65

A Benchmark Survey into the Customer Configuration Processes CHAPTER 4
specific to product software development. One area that is specific to product software
vendors is the fact that they have to release, deliver, and deploy their products on a wide
range of systems, for a wide range of customers, in many variations. Furthermore, these
applications constantly evolve, introducing versioning problems. This chapter presents
a benchmark survey into the customer configuration updating processes of 74 product
software vendors.
To date product software is a packaged configuration of software components or a
software-based service, with auxiliary materials, which is released for and traded in a
specific market [132]. Customer configuration updating is defined as the combination
of the vendor side release process, the product and update delivery process, the
customer side deployment process, and the usage and activation process (see chapter 2).
The release process is how products and updates are made available to testers, pilot
customers, and customers. The delivery process consists of the method and frequency
of update and knowledge delivery from vendor to customer and from customer to
vendor. The deployment process is how a system or customer configuration evolves
between component configurations due to the installation of products and updates.
Finally, the activation and usage process concerns license activation and knowledge
creation on the end-user side.
Vendors encounter many problems when attempting to improve customer
configuration updating. To begin with, these processes are themselves highly
complex considering vendors have to deal with multiple revisions, variable features,
different deployment environments and architectures, different distribution media, and
dependencies on external products (please see chapter 6). Also, there are not many
tools available that support the delivery and deployment of software product releases
that are generic enough to accomplish these tasks for any product (please see chapter 5).
Finally, Customer Configuration Updating (CCU) is traditionally not seen as the core
business of vendors, and seemingly does not add any value to the product, making
vendors reluctant to improve CCU.
This chapter presents the results from a survey of 74 product software vendors. The
respondents are product managers for one product from one product software vendor.
The object under study is the release, delivery, deployment, and usage and activation
processes from each vendor for one of its products. These four CCU processes consist
of two to four practices. Each practice consists of capabilities. A vendor’s capabilities
are measured using between three and five questions per capability. The aims of these
scores are to establish a vendor’s position compared to competitors. The overall goals
of this research are to see what the CCU landscape in the Netherlands looks like,
to establish whether CCU process scores are directly related to product success, and
whether a survey can be used as a knowledge dissemination method.
In the following section the processes and practices of customer configuration
updating are described in detail. In section 4.3 the research design, consisting of
the hypothesis and research approach, is presented. In section 4.4 the survey results
and respondents are presented, including the results for each process area and for the
hypotheses. Finally, in section 4.6 we present our conclusions and discuss our future
work in regards to CCU process improvement.
66

A Benchmark Survey into the Customer Configuration Processes CHAPTER 4
Finally, a vendor’s activation and usage process is based on three practices. First,
a vendor must (semi) automatically handle all license requests and distribute licenses
with a maximum amount of flexibility within the organization. A vendor supports this
practice once customers can explicitly manage their licenses, once licenses expire, once
temporary licenses can be generated for sales and test purposes, and once licenses are
generated automatically as soon as a sales contract is signed. Secondly, vendors must
make use of feedback to gain as much knowledge about the product in the field as
possible. A vendor is considered adequate for this practice once it makes use of both
usage reports and error feedback. The third practice is that vendors must be aware of
their customers’ configurations. Vendors scores for this practice when they are aware
of the software and hardware components that are used by its customers.
4.3 Research Design
This research has been conducted to find out more about the CCU practices and
processes of product software vendors, to benchmark a product software vendor’s
practices, and to generalize some of the conclusions of earlier work we applied in a
multiple case study (please see chapter 2).
4.3.1 Hypotheses
The work has been conducted for two reasons. The first reason is to perform a
benchmark for product software vendors about their release, delivery, deployment, and
usage and activation practices. Secondly, we wished to prove or disprove the following
hypotheses, which have been formulated in earlier research [143]:
H1: A product software vendor’s scores for the CCU processes are positively
correlated to the age of the company, the age of the product, the number of natural
languages in which the product is available, the number of developers working on
the product, and the number of customers of the product. Many different tools are
built around software products during their lifecycle. Furthermore, customers come
and go, making the need for good CCU processes and tools even larger. Also, as a
vendor’s customer base grows, this needs increases further. Also, with more developers
on board, more time can be spent on CCU improvement. We therefore expect H1 to be
true.
H2: A younger technology platform will result into weaker performance of the
CCU processes. When a product software vendor starts using a new technology there
are not many CCU support tools available for that technology. This implies that older
technology platforms will have better CCU support, improving a vendor’s CCU score.
H3: When a vendor explicitly manages CCU knowledge they are more
successful. When a product software vendor changes the CCU process it improves
customer experience and enables a product software vendor to spend less resources
per customer. This frees up resources to further develop the product or do more
maintenance. This hypothesis is two sided, however, due to the fact that a more
successful product software vendor will have more resources available to spend op
CCU processes.
68

A Benchmark Survey into the Customer Configuration Processes CHAPTER 4
4.3.2 Approach and Survey Design
The research hypotheses are proved or disproved using a web survey that establishes
scores of CCU processes and different maturity indicators for product software vendors
and their products. The benchmark survey consists of eleven parts. Each part contains
between three to twelve questions. There are closed as well as open ended questions in
the benchmark survey. The closed questions are used to establish scores for practices
of vendors and consist of yes/no questions and multiple choice questions. The open
questions establish the generic or numeric information on the vendor, such as the
vendor’s name, the amount of customers of the product, the amount of users of a
product, etc. Three open ended questions have been added to the end of the survey
to find out in which parts of CCU the vendor will invest in the future and what tools
they feel are missing in the range of CCU support tools.
Each of the practices stated in Section 4.2 has three to five questions that assesses
the vendor’s practice score. The practices have been derived from the CCU model,
as described in Section 4.2. When all practice scores for one process are added up
we obtain the process score. As is suggested by Saywell [30], methods that force
your public to evaluate their own process actively by participation (in a benchmark
survey, for instance) are a valid method for knowledge dissemination. When sending
out the benchmark report we attached a small paper survey, with a stamped return
envelope, to evaluate Saywells claim. We received 26 responses of which only one was
negative. All others responded positively to the usefulness of the benchmark report as
both a knowledge dissemination tool and as being useful to use in future improvement
projects.
4.3.3 Sample Selection
The respondents have been selected based on 2 criteria: First the submitter must be a
product manager or a development manager who is close to the process and can answer
each question. Secondly, the software product must specifically be a software product
that is delivered to customers and executed at the customer’s site. These requirements
are specified in the invitation e-mail and at the beginning of the benchmark survey
itself.
The vendors have been selected through the Platform for Product Software [45],
the yellow pages, and the Netherlands business index for ICT [84]. The potential
respondents have been approached by e-mail twice. No two respondents belonged
to the same product software company.
4.3.4 Survey Tool
The open source PHP/MySql tool used for this benchmark survey is called PHP
Surveyor [47]. The tool is still under development and the fact that it is open source
has enabled some final customizations, such as the addition of question types and
custom data analysis methods. The survey tool has been selected specifically because
the submitter can save the survey results up to a certain point, to allow him or her to
70

SECTION 4.4 Results
largest exporter of product software in the world [44]. It must be noted that three of
the larger software vendors in the Netherlands replied (in a country where there are
only a handful). Even though we estimate that our dataset covers 6 percent of the total
number of software vendors, we expect this number to be much higher when looking
at the number of employees active in the product software industry.
This work is specific to product software vendors. During the research we were
approached by a number of on-line application service providers, asking whether they
could participate. These requests have been declined, because too many questions
were about concepts that are not applicable to software providers, such as delivery of a
product and its updates, usage and feedback reports.
4.4 Results
In table 4.1 the average process and practice scores are listed for CCU. The first
column listst the process for which this practice is applicable. The second column
describes each practice and provides references to other work in which these practices
are described and discussed. In the third column the average score is provided for
each practice, obtained by the vendors. The final column lists the maximum score that
could have been earnt. The processes and practices of the CCU evaluation are based
on the CCU model [143], the SOFA model [99], and some capabilities of the Software
Dock [52] and have been recorded into the SPICE based evaluation model. Please
note that the total scores for each process cannot be compared to the scores of other
processes. We believe that the four processes are equally important. In tables 4.3, 4.4,
4.5, and 4.6 the average scores for the release, delivery, deployment and usage and
activation practices and processes are listed. The tables describe the questions that are
posed to determine each practice score for the processes. The scores that can be earned
per question have been determined by a committee of five representatives from four
product software vendors. The scores for each practice are determined by between two
and five questions each.
4.4.1 Respondents
74 product managers submitted the survey, from software vendors ranging from 1 to
460 employees (see table 4.2). Three were excluded from the dataset on the grounds
of being from the wrong country or for being a web service provider. Statistics
Netherlands estimates that there are 1400 software vendors in the Netherlands, giving
a 5 percent coverage of the total product software industry. Furthermore, when
asked the product managers to categorize their business by checking at least two
categories (see table 4.2), by far the largest part of product software vendors is building
business productivity tools and enterprise resource planning systems. The product
managers were also asked to provide the development technologies they used (results
in table 4.2). Please note that for the technology and business category questions
respondents were allowed to submit more than one answer.
73

A Benchmark Survey into the Customer Configuration Processes CHAPTER 4
Practice Question or description Average
Score
of all
vendors
Highest
possible
score
A1 How often do you publish a major release of your
product?
5
How often do you publish a minor release of your
product?
5
How often do you publish a bugfix release of your
product?
5
Are releases timed in sync with customer requirements? 2
Total A1: The vendor addresses release package planning
to maintain high quality releases[143]
6.08 17
A2 Does your organisation use a formal release planning
that states the times until the next major, minor, and
bugfix releases?
5
Is this planning published in such a way that all related
personnel can access it?
1
Is there a formal publishing policy towards the outside
world in regards to this document?
1
Total A2: The vendor maintains an explicit release
planning [142]
2.26 7
A3 Is there a step-by-step description (release scenario) of
what happens on the day of a release?
2
Releases are stored in a versioned repository? 6
All major, minor, and bugfix releases can be
downloaded by all product stakeholders?
6
The latest release can be downloaded by all
stakeholders?
4
Total A3: A formalized release scenario that describes
what happens on release day [143]
8.73 18
A4 All tools that are built to support CCU are managed as
if they are externally acquired products.
2
All other tools (such as development tools) are managed
explicitly as well?
2
Are these components stored in a versioned repository? 5
Total A4: Manage external products, such as commercial-
off-the-shelf components [62]
4.73 9
All Release Process 21.8 51
Table 4.3: Scores for Release Practices and Processes
74

SECTION 4.4 Results
Practice Question or description Average
Score
of all
vendors
Highest
possible
score
D1 Do you use license agreements? 5
Can an end-user choose which license he/she will use at
startup?
1
What type of data is stored in your license files? 1
Total D1: Licenses are can be renewed and trialled and
activate components of the software [143]
6.97 7
D2 Can a customer renew its license without your
intervention?
3
Do your licenses expire? 1
Do you often supply temporary licenses? 2
Are licenses generated from contracts automatically? 3
Total D2: Licenses are explicitly managed within the
organization and generated from contracts[143]
2.65 9
D3 Are you aware of how customers use your product? 3
Are you well aware of the hardware customers use? 3
Are error reports automatically sent when a product
crashes?
3
Does your product create usage reports? 2
Are you aware of all customer specific solutions for
your product?
3
Total D3: Vendors know how customers use products and take
advantage of this knowledge [135, 148]
9.24 14
All Usage and activation process 18.86 30
Table 4.6: Scores for Usage and Activation Practices and Processes
77

A Benchmark Survey into the Customer Configuration Processes CHAPTER 4
In regards to deployment the advice “Explicitly manage all relationships between
external products and yours” given 46 times, especially to those vendors who set
“Reduce deployment problems” and “Serve customers more cost-effectively” as their
top priorities. Another improvement suggestion was 46 times as well, but with a lower
average relevance rating, was “Enable your product update tool to facilitate custom
solutions by customers and partners” because many product vendors do not leverage
the advantages of a software supply network yet [152, 151].
Finally, in regards to usage and activation “Send automatic error reports” was
advised 53 times to those software vendors with the top priorities “Reduce deployment
problems” and “Serve customers more cost-effectively”. Furthermore, improvement
suggestions often concerned licenses, such as “Enable the customer to renew their
license without your manual intervention” and “Generate licenses automatically after
entering a sales contract” (43 and 44 times). With the report sent back to the product
software vendors a small survey is included that is used to evaluate the applicability of
the advice.
We received 19 responses from the short survey. All short surveys reported that the
benchmark had changed their view of the CCU processes and that they increasingly
saw the CCU processes as interconnected. Of the 19 surveys two reported that they
planned no new changes based on the benchmark report, two reported that they would
make large changes to their CCU processes, and the other 15 reported they would make
small changes based on the report. The advice was rated per process on a Likert scale.
17 surveys reported that the advice given in three of the four processes were useful. All
recommendations for the processes were rated equally and averaged between useful
and very useful. Some of the open suggestions were that we change the survey for
smaller companies and that we include other subjects such as support and user interface
in our benchmark.
4.4.5 Exploring Relationships
The benchmark survey delivered us with a large set of data, consisting of answers
to yes/no questions, multiple choice questions, and a large amount of open ended
questions. We attempted to find new relationships between yes/no questions using two-
sided Pearson’s goodness of fit chi-square tests. We used an algorithm to find whether
relationships existed between all yes/no questions. We (obviously) found strong
relationships between conditional questions, where the submitter could only provide
an answer if the submitter had answered yes or no to the previous question. Some of
the relations of interest are described below. Please see all the found relationships in
table 4.12.
Many obvious relationships surfaced: first, there exists a relationship between
whether a product’s licenses expire and whether a product software vendor can provide
temporary licenses (r = 16:1141; p  0;0002). Secondly, the ability the update at
runtime and the use of a commercial update tool are related (r = 4:6423; p 0:0313).
Thirdly, the ability to check a customer’s configuration for completeness is related to
the use of components off the shelf that are included in the product (r = 7:0652; p 
0:008). Finally, the capability to send feedback and usage information from customers
to the vendor is related to the expiration of licenses (r = 9:6038; p  0:002), sending
80

A Benchmark Survey into the Customer Configuration Processes CHAPTER 4
most often per floating user or per user name. Pay per usage is quite popular as
well, with 19 of the 74 respondents using it to bill their customers. 45% of the cases
uses licenses that expire. 13 product software vendors have mechanisms in place that
enables customers to renew a license without the intervention of the vendor. 65% of
the cases provide temporary licenses on a regular basis. In 20% of the cases licenses
are automatically generated from contracts.
28% of the vendors make use of usage feedback reports. 78% of the vendors is
aware of all customizations that have been built on top of their products. In 30% of the
cases software vendors send back error reports in case of a crash or error.
The average scores per practice are quite high, which explains that product software
vendors perform well when it comes to license management. Furthermore, they provide
many different reporting features in their products (which are sent back to the vendor
in only 30% of the cases, mind). Product software vendors perform well when it comes
to license management and product usage reporting. This does not necessarily mean
that this area does not deserve attention, though. We believe there is still much research
to be done when it comes to the mining of data from feedback reports.
4.6 Conclusions and Discussion
This chapter presents the results of a survey of 74 product software vendors in the
Netherlands. The survey was created using the CCU process evaluation model, which
is presented in detail. The survey allows us to generalize conclusions from earlier
research that CCU processes of product software companies are generally implemented
to a very limited degree. Furthermore, the results from the survey show that CCU
improvements have a significant positive effect on product success. Finally, the results
demonstrate weak points in CCU processes, such as a lack of CCU tools.
In earlier work the CCU evaluation model was presented [143], and validated at
nine product software vendors. This work, however, could not easily be generalized
to be applicable to a larger group of product software vendors. The survey enabled
a further detailing of the model, adding weights to practices and capabilities using an
expert panel. Furthermore, the survey enables us to validate our hypotheses from the
nine case studies and to show that CCU is an area that urgently requires more research.
The survey results show that CCU is an underdeveloped area that requires more
attention and tooling. The areas that deserve most attention are the release, delivery,
and deployment processes, due to the fact that product software vendors on average
only implement between one third and one half of the capabilities of each practice.
The main results of such research will improve customer retention rates for product
software vendors, product success, and product update and deployment reliability.
CCU research and tools should be geared towards release planning, knowledge
delivery, component updating, and feedback reports. This is indicated by four facts.
First, only 37% of the software vendors uses a formal release planning. Secondly, only
20% of the software vendors use their own product to share knowledge with customers,
such as product knowledge and product news. In the area of deployment many product
software vendors do not perform any configuration correctness checking or provide any
checks before installation of a product. Finally, only 30% use automatic error reporting
84

A Benchmark Survey into the Customer Configuration Processes CHAPTER 4
Question Question
Type
Section
Customers inform us of bugs by: ( An on-line bug system, e-
mail, phone, fax, the product itself automatically sends an error
message to us )
Option
list
Knowledge
Delivery
We inform our customers about the product on at least a: ( daily
basis, weekly basis, monthly basis, 3 monthly basis, yearly basis,
never )
Option
list
Knowledge
Delivery
Our products can be delivered using the following methods:
( our website, CD-ROM, floppy, secure phoneline or internet
connection, e-mail, DVD,USB Stick )
Option
list
Knowledge
Delivery
Our products are delivered using the following mechanisms: (
manual pull, automatic pull, manual push, automatic push )
Option
list
Knowledge
Delivery
Our product’s update facility can be used to download updates
from ANY location
Yes/No Knowledge
Delivery
Is your product delivered to customers on ( a “get-what-you-paid-
for” basis, by sending the full set of components to customers
where a license (file) later deactivates the unpurchased modules?
)
Option
list
Knowledge
Delivery
What types of payment do you use? ( Pay per usage, Pay per
user(name), Pay per time unit, Pay per floating user, Pay for
services, Lump sum )
Option
list
Licensing
Does the license file contain information on: ( The customer name
and address?, The amount of users?, The modules that have been
purchased?, The customers’ server or CPU id?, The customers’
contract id? )
Option
list
Licensing
Can the license be renewed or downloaded by the customer
without intervention from you?
Yes/No Licensing
Do your licenses expire? Yes/No Licensing
Can you provide temporary licenses? Yes/No Licensing
Can the customer manage the license explicitly? Yes/No Licensing
Are licenses generated from contracts automatically? Yes/No Licensing
What tools do you use for the following processes? Please type
p¨roprietaryı¨f you have developed the tool yourself.
String CCU Tool
Support
What tools do you personnally believe are lacking in the industry
for support of the CCU process?
String CCU Tool
Support
How many percent of your deployments are unsuccessful the first
time?
Numeral CCU
generic
What can be done to reduce this number? String CCU
generic
Table 4.9: Survey Part 3
88