Abran, Alain.; Al-Qutaish, Rafa E. and Cuadrado-Gallego, Juan, “Analysis of the ISO 9126 on Software Product Quality
Evaluation from the Metrology and ISO 15939 Perspectives”, WSEAS Transactions on Computers, Vol. 5, No. 11, World
Scientific & Engineering Academy and Society, Greece, 2006, pp. 2778-2786. (ISSN: 1109-2750)
2778
Analysis of the ISO 9126 on Software Product Quality Evaluation
from the Metrology and ISO 15939 Perspectives

ALAIN ABRAN*, RAFA E. AL-QUTAISH*, JUAN J. CUADRADO-GALLEGO**
*École de Technologie Supérieure
University of Québec
1100 Notre-Dame Street West
Montréal, Québec H3C 1K3,
CANADA
** Department of Computer Science
University of Alcalá
28805 Edificio Politécnico
Alcalá de Henares,
SPAIN
alain.abran@etsmtl.ca, rafa.al-qutaish.1@ens.etsmtl.ca, jjcg@uah.es


Abstract: - While metrology has a long tradition of use in physics and chemistry, it is rarely referred to in the
software engineering measurement, and in particular, in the design and documentation of software measures.
Using the ISO 9126-4 Technical Report on the measurement of software quality in use as a case study, this
paper reports on the extent to which this ISO series addresses the metrology criteria typical of classic
measurement. Areas for improvement in the design and documentation of measures proposed in ISO 9126-4 are
identified based on the ISO International Vocabulary of Basic and General Terms in Metrology (VIM) and ISO
15939.


Key-Words: - Metrology, Software Metrics, ISO 15939, ISO 9126, Quality in Use Metrics, Software
Measurement, Software Product Quality.

1 Introduction
In the field of software engineering, the term
“metrics” is used in reference to multiple concepts,
whether in terms of the quantity to be measured
(measurand1), measurement procedures,
measurement results or models of relationships
across multiple measures, or of the objects
themselves. In the software engineering literature,
the term is applied, for instance, to a measure of a
concept (e.g. McCabe cyclomatic complexity [1]),
to quality models (ISO 9126 – software product
quality [2]) and to estimation models (e.g.
Halstead’s equations [3]). This has led to many
curious problems, among them a proliferation of
numerous publications on metrics for concepts of
interest, but with a very low rate of acceptance and
use by either researchers or practitioners, as well as
a lack of consensus on how to validate so many

1 A measurand is defined as a particular quantity subject
to measurement; the specification of a measurand may
require statements about quantities such as time,
temperature and pressure [14].
proposals [4-6]. At the present time, the inventory of
software metrics is so diversified and includes so
many individual proposals [7, 8] that it is not seen to
be economically feasible for either industry or the
research community to investigate each of the
hundreds of alternatives proposed to date.
While metrology has a long tradition of use in
physics and chemistry, it is rarely referred to in the
software engineering literature. Carnahan et al. [9]
are among the first authors to identify this gap in
what they referred to as “IT metrology”; they
highlight the challenges and opportunities arising
from the application of the metrology concepts to
information technology. In addition, they have
proposed logical relationships between metrology
concepts, consisting of four steps to follow to obtain
measured values: defining quantity/attribute,
identifying units and scales, determining the
primary references and settling the secondary
references. Moreover, Gray [10] discusses the
applicability of metrology to information
technology from the software measurement point of
view. Moreover, the ISO in 2002 introduced an
international standard as an international agreement

2779
on measurement terminology for software and
system engineering; that is ISO 15939 [11]. The
ISO 15939 adopted its terminologies from the
International Vocabulary of Basic and General
Terms in Metrology (VIM) [12].
Abran [13] has highlighted some high-level
ambiguities in the domain of software measurement,
and proposed substituting the appropriate metrology
terms for the current ambiguous and peculiar
software metrics terminology unique to the domain
of software engineering. In metrology, the term
“metrics” is never used. In addition, the availability
of the metrology concepts in software engineering
has been investigated in [5, 14, 15]. Abran and
Sellami [6] have documented the metrology
concepts addressed in ISO 19761 (COSMIC-FFP),
both in the design of this measurement method and
in some of its practical uses. Moreover, Sellami and
Abran [16] have investigated the contribution of
metrology concepts to understanding and clarifying
the framework for software measurement validation
proposed by Kitchenham et al. in [17].
The ISO 9126 series of documents on software
product quality evaluation proposes a set of 120
metrics2 for measuring the various characteristics
and subcharacteristics of software quality. However,
as it is typical in the software engineering literature,
their set of so-called metrics in ISO 9126 refers to
multiple distinct concepts which, in metrology,
would have distinct labels (or naming conventions,
e.g. terms) to avoid ambiguities.
To help in understanding and clarifying the
nature of the metrics proposed in ISO TR 9126-4
[18], each is analyzed in this paper from a
metrology perspective and mapped to the relevant
metrology concepts. Such a mapping will also
contribute to identifying the measurement concepts
that have not yet been tackled in the ISO 9126 series
of documents. Each of these gaps represents an
opportunity for improvement in the design and
documentation of the measures proposed in ISO
9126.
This paper presents an overview of the relevant
metrology concepts in section 2, and an overview of
the ISO 9126 series and the quality in use metrics in
Section 3. Section 4 and 5 present the analysis of the
“effectiveness”, “productivity”, “safety”, and
“satisfaction” metrics, respectively. In section 6, a
mapping of the VIM and ISO 15939 terminologies

2 The term “metrics” used in ISO 9126 is replaced by
“measures” in the new series of standards in accordance
with the metrology terminology.
to the quality in use metrics is discussed. A
discussion in section 7 concludes the paper.


2 Metrology
The term “metrology” is defined in the ISO
International Vocabulary of Basic and General
Terms in Metrology as the field of knowledge
dealing with measurement [12]. More specifically, it
has been defined in [19] as: “that portion of
measurement science used to provide, maintain, and
disseminate a consistent set of units; to provide
support for the enforcement of equity in trade by
weights and measurement laws; or to provide data
for quality control in manufacturing”. Following the
above definitions, metrology forms the basis of all
measurement-related concepts in the natural
sciences and engineering, and to each of the
different interpretations of a software metrics is
associated a related distinct metrology term with
related metrology criteria and relationships with
other measurement concepts. In 1984, the ISO, with
other participating organizations (BIPM, IEC and
OIML), published their first edition of the
international consensus on the basic and general
terms in metrology (VIM) [20]. Later, in 1993, this
publication was reviewed, and then the ISO, in
collaboration with six participating organizations
(BIPM, IEC, OIML, IUPAC, IUPAP and IFCC),
published the second edition of this document [12].
The ISO is now working on its third edition of this
document to integrate, in particular, concepts related
to measurement uncertainty and measurement
traceability.
The second VIM edition on metrology presents
120 terms in six categories and in increasing order
of complexity, and describes each term individually
in textual format (in parentheses, the number of
terms in each category): Quantities and Units (22
terms), Measurements (9 terms), Measurement
Results (16 terms), Measuring Instruments (31
terms), Characteristics of the Measuring Instruments
(28 terms) and Measurement Standards – Etalon (14
terms).
To facilitate an understanding of these more than
one hundred related terms, Abran and Sellami [21]
proposed a modeling of all the sets of measurement
concepts documented in this ISO document.
Two of the categories of terms in the VIM deal
with some aspects of the design of measurement
methods, that is, category 1: “quantities and units”,
and category 2: “measurement standards – etalon”.
The other four categories are related to the
application of a measurement design with a
measuring instrument, and to the quality

2780
characteristics of the measurement results provided
by this measuring instrument [21]. More
specifically, we use the first category, which deals
with the design of the measurement methods, that is,
quantities and units and, in particular, the system of
quantities that consists of two types of quantities;
that is, base and derived.3


3 ISO 9126 & Quality in Use Metrics
In 1991, the ISO published its first international
consensus on the terminology for the quality
characteristics for software product evaluation: ISO
9126 – Software Product Evaluation – Quality
Characteristics and Guidelines for their Use [22].
From 2001 to 2004, the ISO published an expanded
four-part version, containing both the ISO quality
models and inventories of proposed measures for
these models. The current version of the ISO 9126
series now consists of one International Standard [2]
and three Technical Reports [18, 23, 24].
The first document of the ISO 9126 series –
Software Product Quality Model – contains two
structures of quality models for software product
quality [2]: a structure for both the internal and
external quality models, and a structure for the
quality in use model. The first structure of the ISO
9126-1 Quality Model includes six characteristics,
subdivided into twenty-seven subcharacteristics for
internal and external quality [2]. These
subcharacteristics are related to internal software
attributes, and are noticeable externally when the
software is used as part of a computer system. The
second structure of the ISO 9126-1 model includes
four quality in use characteristics [2]: effectiveness,
productivity, safety and satisfaction. These
characteristics and subcharacteristics are defined in
the ISO 9126-1 international standard document.
The second document of the ISO 9126 series –
Software Product External Quality Metrics –
contains a set of metrics for each external quality
subcharacteristic, explanations of how to apply and
use them, and examples of how to apply them
during the software product life cycle [23].
The third document of the ISO 9126 series –
Software Product Internal Quality Metrics –
contains an inventory of metrics for each internal
quality subcharacteristic, explanations of the
application of these metrics, and examples of how to
use them in the software product life cycle [24].

3 In ISO 15939, the terms “base quantities” and “derived
quantities” were replaced by equivalent terms: “base
measures” and “derived measures”.
Finally, the fourth document of the ISO 9126
series – Software Product “Quality in Use” Metrics
– contains a basic set of metrics for each quality in
use characteristic, explanations of how to apply
them and examples of how to use them in the
software product life cycle [18].
In ISO 9126-4 [18], fifteen metrics have been
proposed for the quality in use metrics. They have
been classified into four collections of metrics based
on the characteristics presented in ISO 9126-1:
1. Effectiveness: task effectiveness, task
completion and error frequency
2. Satisfaction: task time, task efficiency, economic
productivity, productive proportion and relative
user efficiency
3. Safety: user health and safety, safety of people
affected by use of the system, economic damage
and software damage
4. Productivity: satisfaction scale, satisfaction
questionnaire and discretionary usage
These fifteen metrics are analyzed using a
metrology concept structure from the VIM category,
Quantities and Units [12], based on four
characteristics, that is: system of quantities,
dimension of a quantity, unit of measurement and
value of a quantity.


4 Effectiveness Metrics
In ISO 9126-4, the claim is that the three
Effectiveness Metrics assess whether or not the task
carried out by users achieved the specific goals with
accuracy and completeness in a specific context of
use [18]. This section presents the outcomes of the
mapping of the set of Quantities and Units
metrology concepts to the 2004 description of
Effectiveness Metrics in ISO 9126-4. A summary of
this mapping is presented in the Appendix.

4.1 System of quantities for Effectiveness
4.1.1 Base quantities
Firstly, it can be observed that these three
Effectiveness Metrics are not collected directly by a
measurement system, but are derived from a
computation of four base quantities that are
themselves collected directly, that is: task time,
number of tasks, number of errors made by the user
and proportional value of each missing or incorrect
component.
The first three of these base measures in the
Appendix refer to terms in common use, but this
leaves much to interpretation on what constitutes,
for example, a task: it does not ensure that the
measurement results are repeatable and reproducible

2781
across measurers, across groups measuring the same
software and, as well, across organizations where a
task might be interpreted differently and with
different levels of granularity. This leeway in their
interpretation makes a rather weak basis for either
internal or external benchmarking.
The third base quantity, number of errors made
by the user, is defined in Appendix F of ISO TR
9126-4 as an “instance where test participants did
not complete the task successfully, or had to attempt
portions of the task more than once” [18]. This
definition diverges significantly from the one in the
IEEE Standard Glossary of Software Engineering
Terminology [25] where the term “error” has been
defined as “the difference between a computed,
observed, or measured value or condition and the
true, specified, or theoretically correct value or
condition; for example, a difference of 30 meters
between a computed result and the correct result.”
The fourth base quantity, referred to as the
“proportional value of each missing or incorrect
component” in the task output is based, in turn, on
another definition, whereas each “potential missing
or incorrect component” is given a weighted value
Ai based on the extent to which it detracts from the
value of the output to the business or user [18].
These expansive, embedded definitions contain a
number of subjective assessments for which no
repeatable procedure is provided: the value of the
output to the business or user, the extent to which it
detracts, the components of a task, and potential
missing or incorrect components.

4.1.2 Derived quantities
The proposed three Effectiveness Metrics, which are
defined as a prescribed combination of these base
quantities, are therefore derived quantities. The
ranges of the results obtained from implementing
the corresponding measurement function are
introduced in the upper part of the Appendix for
each of these derived quantities. These quantities
inherit the weaknesses of the base quantities of
which they are composed.

4.2 Dimension of a quantity for Effectiveness
Emerson [26] states that the concept of dimension is
particularly applicable to the derived quantities: two
of them, i.e. task effectiveness and task completion,
can have values between 0 and 1, and would be
considered as dimensionless quantities, since a ratio
of quantities with the same dimensions is itself
dimensionless [26].


4.3 Units of measurement for Effectiveness
The metrology concepts related to units of
measurement are:
-Symbols of the units
-Systems of units
-Coherent (derived) units
-Coherent system of units
-International system of units
-Base units
-Derived units
-Off-system units
-Unit Multiples
-Unit Submultiples
The mappings of these metrology concepts for
Effectiveness Metrics are presented in the Appendix.
Two metrology concepts must be analyzed in more
detail, base units and derived units.

4.3.1 Base units
Of the four base quantities, a single one, i.e. task
time, has an internationally recognized standard
base unit, i.e. the second, or a multiple of this unit. It
also has a universally recognized corresponding
symbol (‘s’). The next two base units (tasks and
errors) do not refer to any international standard of
measurement, and must be locally defined (which
means that they fit poorly, for comparison purposes,
when measured by different people, unless local
measurement protocols have been clearly
documented, and they are implemented rigorously
in a specific organization). The fourth base quantity,
proportional value of each missing or incorrect
component, is puzzling because it is based on a
given weighted value (number) and has no
measurement unit.

4.3.2 Derived units
The derived quantity, task effectiveness, leads to a
derived unit that depends on a given weight (i.e. 1 –
a given weight). Therefore, like the base unit, its
derived unit of measurement is unclear.
The derived quantity, task completion, is
computed by dividing two base quantities (task/task)
with the same unit of measurement.
The definition of the computation of the derived
quantity, error frequency, provides two distinct
alternatives for the elements of this computation.
This can lead to two distinct interpretations, i.e.
errors/task or errors/second. Of course, this in turn
leads to two distinct derived quantities as a result of
implementing two different measurement functions
(formulas) for this derived quantity. Of course, this
leaves open the possibility of misinterpretation and
misuse of measurement results when combined with
other units: for example, measures in centimeters
and measures in inches cannot be added or
multiplied.
This lack of clarity, as well as the lack of
references to international units of measurement,
could explain why there has been no attempt to

2782
integrate the proposed base and derived quantities
into a system of units, including references to
coherent units and a coherent system of units.

4.4 Value of a quantity for Effectiveness
The four types of metrology values of a quantity
are: true value, conventional true value, numerical
value and conventional reference scale.
Numerical values are indeed obtained for each
base quantity based on the defined data collection
procedure; for each derived quantity, the numerical
values are obtained by applying their respective
measurement function. For instance, the derived
quantities, task effectiveness and task completion,
are both percentages, and are interpreted as the
effectiveness and completion of a specific task
respectively.
For task effectiveness in particular, anyone
would be hard pressed to figure out both a true value
and a conventional true value; for task completion
and error frequency, the true values would depend
on locally defined and rigorously applied
measurement procedures, but without reference to
universally recognized conventional true values (as
they are locally defined).
Finally, in terms of the metrological values of a
quantity, only task time refers to a conventional
reference scale, that is, the international standard-
etalon for time, from which the second is derived.
None of the other base quantities in these
effectiveness metrics refers to a conventional
reference scale, or to a locally defined one.


5 Other Quality in Use Metrics
5.1 Productivity Metrics
In ISO 9126-4, the claim is made that the five
productivity metrics assess the resources that users
consume in relation to the effectiveness achieved in
a specific context of use. In this standard, the time
required to complete a task is considered to be the
main resource to take into account [18]. This section
presents the outcome of the mapping of the set of
Quantities and Units metrology concepts to the
2001 description of Productivity Metrics in ISO
9126-4.

5.1.1 System of quantities for Productivity
One of the five proposed productivity metrics in
ISO 9126-4 is a base quantity (task time) while the
other four are derived quantities (task efficiency,
economic productivity, productive portion and
relative user efficiency).
In addition, task efficiency refers explicitly to
another derived quantity, task effectiveness, which
was analyzed in the previous section.
It is to be noted that these derived quantities are
themselves based on five base quantities: task time,
cost of the task, help time, error time, and search
time.

5.1.2 Dimension of a quantity for Productivity
All the productivity metrics, except task time, are
dimensionless quantities.

5.1.3 Units of measurement for Productivity
For the base and derived quantities, there are five
base units and no explicit derived units. However, it
can be observed that the measurement unit for task
effectiveness is not completely clear, since it
depends on an ill-defined “given weight”:
.
ondsec
unit ess'effectiventask '
=unit 'efficiencytask '
.
second
?
=
second
unit ht'given weig a' -1
= (1)
Similarly, the measurement unit of economic
productivity depends on the measurement unit of
task effectiveness, a derived quantity which its unit
of measurement is unknown:

unitcurrency
unit ess'effectiventask '
=unit ty'productivi economic'

unitcurrency
unit ht'given weig a' -1
=
.
unitcurrency
?
= (2)
Since there is no measurement unit for the
productive proportion (it has the same measurement
unit in both the numerator and the denominator), the
result is a percentage:
.
second
second
=unit 'proportion productive' (3)
Finally, for relative user efficiency, there is no
measurement unit either, since the measurement
units in both the numerator and the denominator are
the same here as well (the task efficiency
measurement unit), and therefore the result of this
derived quantity is also a percentage. This point can
be clarified as follows:
unit 'efficiencytask '
unit 'efficiencytask '
=unit 'efficiencyuser relative'

second
unit ess'effectiventask '

second
unit ess'effectiventask '

=

second
unit ht'given weig a' -1

second
unit ht'given weig a' -1

= .
second
?

second
?

= (4)

2783

5.2 Safety Metrics
In ISO 9126-4, the safety metrics claim to assess the
level of risk of harm to people, businesses, software,
property or the environment in a specific context of
use; their scope includes the health and safety of
both the users and those who affected by use, as
well as unintended physical or economic
consequences [18].
To evaluate the safety characteristics of a
software product, four derived quantities must be
quantified (i.e. user health and safety, software
damage, economic damage and the safety of people
affected by use of the system). Each of these derived
quantities is the result of a computational formula
(function), which consists of a combination of pre-
collected base quantities (i.e. number of usage
situations, number of people, number of occurrences
of software corruption, number of occurrences of
economic corruption and number of users). It can be
observed that the resulting values of all the derived
quantities should be between 0 and 1.
All the safety metrics are dimensionless
quantities; there are five base units and two derived
units for these quantities. In addition, two of the
derived quantities have no measurement units, since
the measurement unit is the same in both the
numerator and the denominator, i.e. user health and
safety and safety of people affected by use of the
system, whereas none of the measurement units has
a symbol.

5.3 Satisfaction Metrics
The satisfaction metrics in ISO 9126-4 claim to
assess the user’s attitudes towards the use of the
product in a specific context of use [18].
All three proposed satisfaction metrics are
derived quantities (i.e. satisfaction scale, satisfaction
questionnaire and discretionary usage), which
themselves depend on four base quantities (i.e.
population average, number of responses, number of
times that specific software function / application /
systems are used and number of times that specific
software function/application/systems are intended
to be used). Two of the proposed satisfaction
metrics are dimensionless quantities, i.e. satisfaction
questionnaire and discretionary usage.
Regarding the measurement units, there are four
base units and no derived units; however, the
measurement unit, satisfaction scale, is not clear,
since it depends on a “questionnaire producing
psychometric scales”. The clarification of this point
is as follows:
.
people
unit scale icpsychometr
=unit scale'on satisfacti' (5)


6 Mapping Terminology with VIM
and ISO 15939
While the term “metrics” is used in ISO 9126, the
use of this term will be abandoned and replaced by
“measures” in the next ISO version currently in
preparation (ISO 25000) as an initial step towards
harmonizing the software engineering measurement
terminology with the metrology guide (VIM) and
ISO 15939 terminologies. Thus, throughout this
section, the term “measures” will be used instead of
the terms “metrics” and “quantities”.
The ISO 9126 internal quality, external quality,
and quality in use sets of measures could be
classified into derived or base measures. The
derived and base measures have been defined in
VIM [12] and ISO 15939 [11] as the following:
- Base measure (metric): the measure that is
defined in terms of an attribute and the method
for quantifying it.
- Derived measure (metric): the measure that is
defined as a function of two or more values of
base measures.
In practice, the data collection associated with a
property of an object (or concept), and
quantification of it, happens at the base measure
level, at which time a measurement unit is assigned
based on the rules of the measurement method used
for the quantification.
At the derived measure level, the base measures
have been already collected and are being
assembled according to the combination rules (e.g. a
computational formula) defined within each derived
measure. A derived measured is therefore the
product of a set of measurement units properly
combined (through a measurement function). This
combination is then labeled to represent an attribute
(of a characteristic or subcharacteristic of the
quality) of a software product.
Table 1 shows a list of the base measures which
are used in the definitions of the measures
documented in ISO 9126-4 [18]. For each of these
measures, this table shows the name of each base
measure, the unit of measurement that is given to its
value, and where each of these base measures could
be used throughout ISO 9126-4. These base
measures can then be used to calculate each of the
derived measures in ISO 9126-4.
Each of these base measures must be collected
individually. They can be used at least once, or
multiple times, for obtaining the derived measure

2784
required to quantify the software properties
specified in the ISO 9126 quality model. For
instance, the base measure, “task time”, is used only
once in measuring the “satisfaction” characteristic,
while the base measure, “number of people
potentially affected by the system”, can be used in
measuring 2 characteristics of quality in use; that is,
“safety” and “productivity”. The construction of
derived measures is based on a computational
formula consisting of two or more base measures.
Such lists of base measures and of the usage
cross-references are currently missing from ISO
9126 and would be helpful to those designing
programs for implementing measurement of the
quality of software products using ISO 9126 quality
models and related measures. In particular, these
lists can help in:
- Identifying, selecting and collecting a base
measure (once), and then using this base measure
to evaluate a number of derived measures.
- Knowledge of which base measures are required
to evaluate specific software quality attributes
(characteristics and subcharacteristics).

Table 1: Cross-Reference of ISO 9126-4 Base Measures Usages
Metrics Name Measurement Unit Effectiveness Satisfaction Safety Productivity
1 Number of Tasks Task 3
2 Task Time Second 3
3 Cost of the Task Dollar 3
4 Help Time Second 3
5 Error Time Second 3
6 Search Time Second 3
7 Number of Users User 3
8 Number of People Potentially Affected by the System Person 3 3
9 Number of Usage Situations Situation 3
10 Number of Errors Made by the User Error 3
11 Proportional Value of each Missing or Incorrect None!! 3


7 Conclusions
The ISO International Vocabulary of Basic and
General Terms in Metrology (VIM) represents the
international consensus on a common and general
terminology of metrology concepts. However, until
recently, it was not usual practice in software
engineering measurement to take into account
metrology concepts and criteria, either in the design
of software measures or in their use and in the
interpretation of measurement results.
This paper has presented an analysis of the ISO
9126-4 Technical Report on quality in use metrics,
and has investigated the extent to which it addresses
the metrology criteria found in classic measurement.
Based on the analysis in sections 4 to 6, the
following comments and suggestions can be made:
- Identifying and classifying the quality in use
metrics into base and derived quantities makes it
easy to determine which should be collected
(base quantities) to be used subsequently in
computing the other (derived) quantities.
- Based on equations (1) and (3 to 5), some of the
derived units are ambiguous, since they depend
on other quantities with unknown units.
- None of the current quality in use metrics refers
to any system of units, coherent (derived) unit,
coherent system of units, international system of
units (SI), off-system units, multiple of a unit,
submultiple of a unit, true values, conventional
true values or numerical values.
- None of the base and derived measures
(quantities), except for task time, has symbols for
their measurement units.
It is to be noted that the ranges of the results of
many of the derived measures in ISO 9126-4 are
between 1 and 0. Therefore, it is easy to convert
them to percentage values. However, from our point
of view, these results would be easier to understand
if they were ranked in terms of qualitative values;
for example, for task completion, if the percentage
result is 100%, then the completion of the task is
labeled “excellent”; if the result is 80%, then the
completion of the task is labeled “very good”; and
so on.

2785
Using the ISO 9126-4 Technical Report on the
measurement of software quality in use as a case
study, this paper has investigated and reported on
the extent to which this ISO series addresses the
metrology criteria typical of classic measurement.
Areas for improvement in the design and
documentation of measures proposed in ISO 9126
have been identified. The analysis methodology
developed to investigate ISO 9126-4 technical
report could also be of use to analyze the
metrological strengths and weaknesses of close to
120 metrics proposed by the ISO in 9126-2 and -3
technical reports.

References:
[1] McCabe, T. J., "A Complexity Measure," IEEE
Transaction on Software Engineering, Vol. 2,
No. 4, 1976, pp. 308-320.
[2] ISO/IEC, ISO/IEC 9126-1: Software
Engineering - Product Quality - Part 1:
Quality Model, International Organization for
Standardization, Geneva, Switzerland, 2001.
[3] Halstead, M. H., Elements of Software Science,
Elsevier North-Holland, New York, 1977.
[4] Jacquet, J. P. and Abran, A., "Metrics
Validation Proposals: A structured Analysis",
8th International Workshop on Software
Measurement, Magdeburg, Germany, 1998.
[5] Abran, A. and Sellami, A., "Measurement and
Metrology Requirements for Empirical Studies
in Software Engineering", 10th International
Workshop on Software Technology and
Engineering Practice (STEP), Montreal,
Canada, 2002.
[6] Abran, A. and Sellami, A., "Analysis of
Software Measures Using Metrology Concepts
- ISO 19761 Case Study", International
Workshop on Software Audits and Metrics
SAM’2004, 6th International Conference on
Enterprise Information Systems ICEIS 2004,
Porto, Portugal, 2004.
[7] Kaner, C. and Bond, W. P., "Software
Engineering Metrics: What Do They Measure
and How Do We know?", 10th International
Software Metrics Symposium (METRICS
2004), Chicago, Illinois, USA, 2004.
[8] Zuse, H., A Framework of Software
Measurement, Walter de Gruyter, Berlin,
Germany, 1998.
[9] Carnahan, L., Carver, G., Gray, M., Hogan, M.,
Hopp, T., Horlick, J., Lyon, G., and Messina,
E., "Metrology for Information Technology",
StandardView, Vol. 5, No. 3, 1997, pp. 103-
109.
[10] Gray, M. M., "Applicability of Metrology to
Information Technology," Journal of Research
of the National Institute of Standards and
Technology, Vol. 104, No. 6, 1999, pp. 567-
578.
[11] ISO/IEC, ISO/IEC IS 15939: Software
Engineering - Software Measurement Process,
International Organization for Standardization,
Geneva, Switzerland, 2002.
[12] ISO/IEC, International Vocabulary of Basic
and General Terms in Metrology (VIM),
International Organization for Standardization,
Geneva, Switzerland, 1993.
[13] Abran, A., "Software Metrics Need to Mature
into Software Metrology (Recommendations)",
NIST Workshop on Advancing Measurements
and Testing for Information Technology (IT),
Maryland, USA, 1998.
[14] Abran, A., Sellami, A., and Suryn, W.,
"Metrology, Measurement and Metrics in
Software Engineering", 9th International
Software Metrics Symposium, Sydney,
Australia, 2003.
[15] Bourque, P., Wolff, S., Dupuis, R., Sellami, A.,
and Abran, A., "Lack of Consensus on
Measurement in Software Engineering:
Investigation of Related Issues", 14th
International Workshop on Software
Measurement (IWSM), Magdeburg, Germany,
2004.
[16] Sellami, A. and Abran, A., "The contribution
of metrology concepts to understanding and
clarifying a proposed framework for software
measurement validation", 13th International
Workshop on Software Measurement (IWSM),
Montreal, Canada, 2003.
[17] Kitchenham, B., Pfleeger, S. L., and Fenton,
N., "Towards a Framework for Software
Measurement Validation," IEEE Transaction
on Software Engineering, Vol. 21, No. 12,
1995, pp. 929-944.
[18] ISO/IEC, ISO/IEC TR 9126-4: Software
Engineering - Product Quality - Part 4:
Quality in Use Metrics, International
Organization for Standardization, Geneva,
Switzerland, 2004.
[19] Simpson, J. A., "Foundations of Metrology,"
Journal of Research of the National Bureau of
Standards, Vol. 86, No. 3, 1981, pp. 36-42.
[20] ISO/IEC, International Vocabulary of Basic
and General Terms in Metrology (VIM),
International Organization for Standardization,
Geneva, Switzerland, 1984.
[21] Abran, A. and Sellami, A., "Initial Modeling of
the Measurement Concepts in the ISO

2786
Vocabulary of Terms in Metrology", 12th
International Workshop on Software
Measurement (IWSM), Magdeburg, Germany,
2002.
[22] ISO/IEC, ISO/IEC IS 9126: Software Product
Evaluation - Quality Characteristics and
Guidelines for Their Use, International
Organization for Standardization, Geneva,
Switzerland, 1991.
[23] ISO/IEC, ISO/IEC TR 9126-2: Software
Engineering - Product Quality - Part 2:
External Metrics, International Organization
for Standardization, Geneva, Switzerland,
2003.
[24] ISO/IEC, ISO/IEC TR 9126-3: Software
Engineering - Product Quality - Part 3:
Internal Metrics, International Organization for
Standardization, Geneva, Switzerland, 2003.
[25] IEEE, Std. 610.12-1990: Standard Glossary of
Software Engineering Terminology, the
Institute of Electrical and Electronics
Engineers, New York, USA, 1990.
[26] Emerson, W. H., "Short Communication on the
Concept of Dimension," Metrologia, Vol. 42,
No. 2, 2005, pp. 21–22.



Appendix: “Quantities and Units” Metrology Concepts in
ISO 9126-4 – Effectiveness Metrics
Metrology Concepts ISO 9126-4 (Effectiveness Metrics*)
5.1 System of Quantities:

- Base Quantities:

1. Task Time.
2. Number of Tasks.
3. Number of Errors Made by the User.
4. Proportional Value of each Missing or Incorrect
component.

- Derived Quantities: 5. Task Effectiveness*. 0 ≤ Task Effectiveness ≤ 1
6. Task Completion*. 0 ≤ Task Completion ≤ 1
7. Error Frequency*. Error Frequency ≥ 0
5.2 Dimension of a Quantity:
- Dimensionless Quantities: 5. Task Effectiveness.
6. Task Completion.

5.3 Units of Measurement:
- Symbols of the Units: - s (Second)
- Systems of Units: - None.
- Coherent (Derived) Units: - None.
- Coherent System of Units: - None.
- International System of Units
(SI): - None.
- Base Units: 1. Second. 2. Task. 3. Error. 4. None (ill-defined)
- Derived Units:
5. (1- a given weight). 6. Task/Task = % 7. Error/Task or Error/Second.
- Off-System Units: - None.
- Multiple of a Unit: - None.
- Submultiple of a Unit: - None.
5.4 Value of a Quantity:
- True Values: - None.
- Conventional True Values: - None.
- Numerical Values:

- Results of applying the measurement functions of the
above base and derived quantities.
- Conventional Reference Scales: 1- Task Time.