Measure Software – and its Evolution – Using Information
Content
Tom Arbuckle
Electronic and Computer Engineering
University of Limerick
Ireland
tom.arbuckle@ieee.org
ABSTRACT
To be able to examine software evolution – variation in soft-
ware over a sequence of releases – or to compare differing
versions of software with each other, we need to be able to
measure artefacts representative of the software or its cre-
ation process. One can find in the literature a multitude
of approaches to both measuring software – by defining and
applying software metrics – and to examining software evo-
lution in terms of these metrics. In this position paper,
we claim that information content, specifically the (relative)
Kolmogorov complexity, is the correct and fundamental tool
for the measurement of software artefacts. Experimental re-
sults obtained from an analysis of the project udev demon-
strate utility: future work should explore the breadth of
applicability and determine the full scope of the approach.
Categories and Subject Descriptors
D.2.8 [Software Engineering]: Metrics—complexity mea-
sures, performance measures; D.2.7 [Software Engineer-
ing]: Distribution, Maintenance, Enhancement; H.1.1 [Models
and Principles]: Systems and Information Theory
General Terms
Design, Documentation, Measurement, Theory
Keywords
Kolmogorov complexity, software measurement, software evo-
lution, information theory, similarity metric, CompLearn
1. INTRODUCTION
This paper’s thesis is that, fundamentally, software arte-
facts of all types can best be measured and compared in
terms of their information content as measured by their Kol-
mogorov complexity.
If we wish to measure or investigate how software evolves
– how subsequent releases of a software project differ from
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, to
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee.
IWPSE-Evol’09, August 24–25, 2009, Amsterdam, The Netherlands.
Copyright 2009 ACM 978-1-60558-678-6/09/08 ...$10.00.
each other – we need to have some means of measurement
by which we can reveal the course of the evolution.
In software engineering (hereafter SE) we measure soft-
ware using software metrics. Software metrics are (abstract)
tools that we apply to artefacts representative of the soft-
ware or of its production process to produce numerical rep-
resentations. In turn, we hope that these numbers represent
what we do when we create software and thereby provide us
with the information we need to measure and predict.
Here we take the word ‘artefact’ to mean any product or
by-product of the software creation process that we can then
use as input to one of our measurement tools. In studying
software evolution, we apply metrics to the artefacts taken at
given points in time or corresponding to different, generally
sequential, releases of the software project. In addition to
metrics that we can apply to single data sets (releases) we
can, alternatively, look for metrics that explicitly involve
either the process of creation or distinctions between these
different data sets.
There are very many software metrics [18, 27, 15, 10, 20,
35, 14]. There are many artefacts that have been considered
for the measurement of software evolution [21].
We have two main reasons for claiming that Kolmogorov
complexity is the ‘best’ measurement device for software
artefacts. Firstly, the Kolmogorov complexity is a funda-
mental mathematical concept which is, by definition, a mea-
surement of the number of bits of information intrinsic to an
object. Being fundamental and having a mathematical def-
inition, it is unlikely that there will be significant disagree-
ment about its interpretation (of and by itself). Secondly,
its fundamental nature means that it involves no tunable pa-
rameters, no expert evaluations, no calibration. Therefore
it can and should serve as a basis and as an underpinning
for other derived metrics extant in the literature that have
already been shown to have practical worth and, in so doing,
should provide an unequivocal means of comparison.
There are other reasons behind our claim – and we are
also aware of some criticisms of it. These will be discussed
in the following sections. We will continue to expound on
our claim, showing its relation to previous work, in the next
section. In the third section of this paper, we will provide
some evidence of the validity of the claim by showing how
the (relative) Kolmogorov complexity can be employed in a
short proof-of-concept example. We study the open source
project udev [24]. In our conclusion, as well as summing up,
we will briefly discuss some further potential opportunities
for application of the method.
129

2. EXPOSITION
We claim that a theoretical basis for the measurement of
software will come from the field of information theory, that
it will be based on Kolmogorov complexity, and that relative
measures of Kolmogorov complexity are already available
and of practical significance. The reasoning behind these
claims is now dealt with in detail.
2.1 Background
We can define any software metric that seems useful to
us – and many people have done so. When considering soft-
ware evolution, the class of potential metrics is broader still.
Practitioners have derived some utility from at least some
of the proposed software metrics. Through ad hoc rules-of-
thumb and heuristics, they have sought to bring some order
to the practice of SE by deriving design advice from numbers
in some way representative of software. Software, in turn,
reflects the decisions made in its design and construction.
Michael Jackson [19] has argued that SE has not yet spe-
cialised to the extent that software construction can be con-
sidered the purview of ‘normal design’ as opposed to ‘radical
design’. Indeed, in cases where software construction can be
reduced to normal design (the minor or automated modifica-
tions of existing designs) risk is minimised and predictabil-
ity is higher. Many, if not all, of the software metrics in use
see greatest success when used within the ‘normal design’
paradigm, areas where the metrics are known to be ‘good’.
However, despite strong arguments advocating the con-
trary, it is still the case that much of software development is
radical rather than normal. Practitioners’ battery of heuris-
tics can, as a result, provide only guidelines and cannot be
more finely tuned to software measurement or the revela-
tion of software evolution without losing their generality. If
there were some underlying theoretical approach that could
justify and validate the measurement of software, it could
instead serve both as a basis for future modelling and as a
common thread, tying previous indicators together.
Information theory is the branch of science that we have
for reasoning about information: if we agree that software
is information, then a theoretical basis for measuring and
comparing software will come from this field. This assertion
is validated by contemporary work, such as that by Harman
[16, 17] – describing the need for information theoretic met-
rics as fitness functions in search-based SE with Lutz [26]
as an exemplar – or that by Clark et al. [13] or by McCa-
mant and Ernst [28] – on information flow with implications
particularly for software security. It was also the conclusion
of early workers in SE. van Emden’s theoretical work [33,
34] extending themes from Simon and Ando [31] and from
Alexander [1] and the practical demonstrations of employing
entropy for measurement of software design by Chanon [8, 9]
were the first applications of information theory in the new
field of SE. These initial forays were followed by many pa-
pers seeking to use variants of entropy to measure software
or its design or its evolution [35, 22].
Consider the discrete case of Shannon’s entropy [30]:
H(X) = −
X
xǫX
px log px. (1)
The entropy, H(X), is calculated in terms of the probabil-
ities px that the codes x from a set X will occur during
their transmission from a source to a sink over a channel.
We can ask: in SE terms, what are the symbols here, what
do they imply, and how do we measure their probabilities
of occurrence? Clearly, given the significant numbers of pa-
pers seeking to employ entropy in SE, there are many possi-
ble answers. Alternatively, we can try a different approach.
Shannon’s entropy relates to the significance of occurrence
of symbols taken from a known alphabet1 but says nothing
about the information content of the symbols themselves.
The information content of the symbols is measured using
Kolmogorov complexity.
2.2 Kolmogorov Complexity
Kolmogorov complexity, also known as “algorithmic en-
tropy”, was discovered independently by Solomonoff [32],
Kolmogorov [23] and Chaitin [7]. The Kolmogorov complex-
ity, K(x), of a binary string x is, by definition, the length of
the shortest (prefix-free) binary program to compute x on a
universal computer, such as a universal Turing machine. It
gives the number of bits to computationally describe x.
Kolmogorov complexity complements entropy. Shannon’s
concern centres on the characterisation of messages from a
random source. Kolmogorov dispenses with probability and
considers only individual messages’ information content. For
the purposes of software measurement, we are claiming that
Kolmogorov complexity provides a more appropriate fit.
There is, however, a practical difficulty to this. As a
definition based on an abstract universal computing engine
(the Turing machine), one might wonder how the value of
Kolmogorov complexity can be found. Indeed, as a non-
partial recursive function, the Kolmogorov complexity is
non-computable. Some form of approximation is needed.
Note firstly, that Kolmogorov complexity and Shannon
entropy can be related. It can be shown that the expected
Kolmogorov complexity for any distribution will be close to
its Shannon entropy. The approximation of the Kolmogorov
complexity in terms of entropy is also the device used by
Allen et al. [2]. They show that
E(Len(x)) ≥ H(x) (2)
where, for an instantaneous code for the domain of x, the
expected length per item is E(Len(x)). They explain that
“the entropy of a distribution of x is the minimum expected
length of an instantaneous code for one sample item”. More-
over, by using a Shannon-Fano coding of n items from a set
X, members x, the Kolmogorov complexity of the set X can
then be written ([2], p.189)
bK(X) = nH(x). (3)
By employing graph representations of programs, taking el-
ements from closed sets thus fixing n, they can then use
counting arguments to derive H(x) and from that bK(X).
One interesting conclusion of their paper is that “Informa-
tion size is highly correlated with counting size”. Given that
many SE metrics count features representative of software
artefacts - lines, methods, calls - we claim that this result
provides some evidence both for our argument but also for
those who may claim that existing metrics are good enough.
Now we know why.
2.3 Information Distance, Relative Measures
Our interest is in monitoring change over software releases,
so we are interested in means of comparison. Kolmogorov
1What is the alphabet for radical design?
130