Position Papers 35
Harmonization Issues in the Updating of ISO Standards
on Software Product Quality

Alain Abran, Rafa E. Al-Qutaish, Jean-Marc Desharnais

École de Technologie Supérieure (ÉTS)
University of Québec
1100 Notre-Dame Street West
Montréal, Québec H3C 1K3, Canada

alain.abran@etsmtl.ca, rafa.al-qutaish.1@ens.etsmtl.ca,
jean-marc.desharnais@etsmtl.ca


Abstract. Within the context of the current ISO project to upgrade the set of technical
reports on the measurement of the quality of software products (ISO 9126), the ISO
working group concerned has come up with proposals for various documents
(standards or technical reports) in the new ISO 25000 series to improve the
interpretation and use of the quality measures. This paper investigates some of the
harmonization issues arising with the addition of new documents like ISO 25021, in
particular with respect to previously published measurement standards for software
engineering, such as ISO 9126, ISO 15939, ISO 14143-1 and ISO 19761.

Keywords: Software Product Quality, Software Measurement, ISO 25021, ISO 9126,
ISO 15939.


1 Introduction

In 1991, the ISO published its first international consensus on the terminology for the
quality characteristics for software product evaluation (ISO 9126:1991) [1]. During the
period 2001 to 2004, the ISO published an expanded version, containing the ISO
quality models and a consensus on inventories of proposed measures for these
models. The current version of the ISO 9126 series of standards consists of four
documents [2-5]:

- ISO 9126-1: Quality Models
- ISO TR 9126-2: External Metrics1
- ISO TR 9126-3: Internal Metrics
- ISO TR 9126-4: Quality in Use Metrics

The ISO has now recognized a need for further enhancements to ISO 9126, primarily
as a result of advances in the field of information technologies and changes in
environment [6]. Therefore, the ISO is now working on the next generation of
software product quality standards, which will be referred to as Software Product

1 The term ‘metrics’ used in ISO 9126 is replaced by ‘measures’ in the new series of standards, in
accordance with ISO 15939.
Abran, Alain; Al-Qutaish, Rafa E. and Desharnais, Jean-Marc, “Harmonization Issues in the Updating of
the ISO Standards on Software Product Quality”, Metrics News Journal, Vol. 10, No. 2, Otto-von-Guericke
University of Magdeburg, Germany, December, 2005. pp. 35-44. (ISSN: 1431-8008)

Position Papers 36
Quality Requirements and Evaluation (ISO 25000). This series of standards will
replace the current ISO 9126 and ISO 14598 series, and will consist of five divisions
[7], each of which may contain one or more documents:

- ISO 2500n: Quality Management Division
- ISO 2501n: Quality Model Division
- ISO 2502n: Quality measurement Division
- ISO 2503n: Quality Requirements Division
- ISO 2504n: Quality Evaluation Division

This work is being carried out by Working Group 6 (WG6) of the software and system
engineering subcommittee (SC7) of the ISO/IEC joint technical committee (JTC1) on
Information Technology, that is, ISO/IEC JTC1/SC7.

One of the objectives of this new ISO 25000 series (and what makes it different from
the current ISO 9126 series) is the harmonization of its contents with the software
measurement terminology of ISO 15939 [8], itself based on the ISO metrology
terminology [9]. Figure 1 shows the proposed structure of the quality measurement
division (ISO 2502n) series that is to replace the current four-part ISO 9126 series of
standards [10]. This proposed quality measurement division (ISO 2502n) will consist
of five documents:

- ISO 25020: Measurement Reference Model and Guide
- ISO 25021: Quality Measure Elements
- ISO 25022: Measurement of Internal Quality
- ISO 25023: Measurement of External Quality
- ISO 25024: Measurement of Quality in Use

Figure 1: WG6 Proposed Structure of the Measurement Division (ISO 2502n series)
Quality Measure Elements (ISO 25021)
Measurement of External
Quality (ISO 25023)
Measurement of Internal
Quality (ISO 25022)
Measurement of Quality
in Use (ISO 25024)
Measurement Reference Model and Guide (ISO
25020)

Included in this new set of technical reports is a proposed new structure with
additional new concepts, such as: ‘quality measure elements’ (QME) and ‘software
quality measures’ [10]. This paper also investigates these proposed concepts, their

Position Papers 37
use and interpretation, and their relationship to similar concepts in other ISO
documents.

This paper discusses the issues concerning terminology harmonization in section 2,
and the issues concerning the harmonization of quality model coverage between ISO
DTR 25021 and ISO 9126 in section 3. A discussion, conclusions and
recommendations are presented in section 4.

2 Terminology harmonization

2.1 Metrology terminology

The ISO 9126 working group (WG6) has proposed the introduction of four new
expressions in ISO DTR 25021 [10], namely: ‘Quality Measure Elements’, ‘General
Quality Measure Elements’, ‘Specific Quality Measure Elements’ and ‘Quality
Measures’. The introduction of these new terms raises the following concern: either
the proper mapping to the set of classic metrology terms has not yet been completed
or there are concepts and related terms missing in the metrology vocabulary. The
latter would be surprising, since metrology is a rather mature domain of knowledge
based on centuries of expertise in the field of measurement and related international
standardization. In this paper, we revisit the WG6 proposal in order to recommend a
proper mapping of concepts to the related metrology [9] terms and to ISO 15939 [8].
The following two expressions come from the ISO standard on software
measurement process, ISO/IEC 15939 [8], which is itself based on the definitions in
the ISO International Vocabulary of Basic and General Terms in Metrology (VIM
1993) [9]:

Base measure: a measure defined in terms of an attribute and the method for
quantifying it. A base measure is functionally independent of other
measures.

Derived measure: a measure defined as a function of two or more values of base
measures. A transformation of a base measure using a
mathematical function can also be considered as a derived
measure.

In [10], it is claimed that a quality measure element is either a base measure or a
derived measure, but then the consensual metrology terms are ignored in favor of
locally defined WG6 measures, thus bypassing the ISO and SC7 harmonization
requirements on measurement terminology.

The ‘quality measure elements’ are described as an input for the measurement of the
‘software quality measures’ of external quality, internal quality and quality in use [10].
Figure 2 shows the proposed relationship between the ‘Quality Measure Elements’
and the ‘Software Quality Measures’, and between the ‘Software Quality Measures’
and the quality characteristics and subcharacteristics. In metrology, these would
correspond to base measures and derived measures respectively. It can be observed
as well that these measures, in particular the derived measures, are defined
specifically to measure the subcharacteristics of internal and external quality or the

Position Papers 38
characteristics of quality in use. None of these is directly related to the top level of
‘software quality’ (which is itself decomposed into three models, then into 16
characteristics and further into a large number of subcharacteristics). Therefore, the
expression selected, in [10], ‘software quality measures’,, is at a level of abstraction
that does not represent the proper mapping of the measures to the concept being
measured.

Figure 2: Quality Measure Elements Concept in the "Software Product Quality
Measurement Reference Model" [10]

composed of

composed of
indicate
indicate

Software Product Quality
Quality
Characteristics

generate

Software Quality
Measures

are applied
t
Measurement
Function

Quality Measure
Elements
Quality
Sub-Characteristics


2.2 Harmonization with the ISO 15939 Information Model

The ISO 15939 information model has been divided into three different sections: data
collection, data preparation and data analysis [11]. Figure 3 shows a mapping
between this information model (left-had side of Fig. 3) and the software product
quality measurement and evaluation of ISO series 9126 and 14598 (right-hand side
of Fig. 3).

2.3 Description Harmonization

The WG6 proposal in [10] recommends next a set of 15 ‘General Quality Measure
Elements’ – Table 1 – to be used as ‘Specific Quality Measure Elements’ within the
software product life cycle; [10] includes a description of its selection of 61 such
‘Specific Quality Measure Elements’. However, it is noted that there are no specific
quality measure elements related to the general quality measure elements ‘Number
of User Operations’ or ‘Number of System Operations’.

Position Papers 39
Figure 3: Mapping between ISO 15939 Information Model and ISO 9126 and ISO 14598


Information Product
Data Analysis

Data Preparation

Analysis Model
Indicator
Derived Measure
Measurement Function
Data Collection
Base Measure
Measurement Method
Attributes
Interpretation
ISO : Software Product Evaluation
- ISO 14598-1: General overview.
- ISO 14598-2: Planning and management.
- ISO 14598-3: Process for developers.
- ISO 14598-4: Process for acquirers.
- ISO 14598-5: Process for evaluators.
- ISO 14598-6: Documentation of evaluation modules.
ISO Software Product Quality Models
- ISO 9126-1 Internal and External Quality Characteristics:
1. Functionality. 2. Reliability. 3. Usability.
4. Efficiency. 5. Maintainability. 6. Portability.
- ISO 9126-1 Quality in Use Characteristics:
1. Effectiveness. 2. Productivity. 3. Satisfaction. 4. Safety.
ISO - Software Product Quality Derived Measures
- In ISO TR 9126, parts 2 to 4, the Internal Quality and External
Quality subcharacteristics and Quality in Use characteristics
each have a number of derived measures.
- There are 27 subcharacteristics each for Internal and External
Software Product Quality.
ISO – Base Measures
In TR 9126 parts 2 to 4, the base measures are labeled and
used in ISO TR 9126, parts 2 to 4, but they are not defined nor
described at a detailed level.

For the description of each of these quality measure elements, different ‘aspects’ are
proposed in [10]:

1. Scale type: the aspect related to the scale type used for measurement.
2. Focus: the aspect related to the scope and objective of the measurement (e.g.
the software product itself, the software product in a system, the software
product in a system used by a specified user in a specified scenario).
3. Method type: the aspect related to the measurement method type relating to
the quality measure element used for measurement.

For aspect 3, it is stated in the same document that the scope and objective are
related to the different parts of ISO 9126 (internal quality, external quality and quality
in use). It must be noted that the use of expressions such as ‘scope’ in [10] for a
measurement method is not harmonized with the corresponding ‘scope’ terminology
used in other ISO software measurement-related standards, such as ISO 14143-1
[12] and ISO 19761 [13].

Position Papers 40
Table 1: ‘General Quality Measure Elements’ [10]

1. Number of Functions 2. Number of Failures
3. Number of Faults 4. Product Size
5. Time Duration 6. Number of Test Cases
7. Number of Restarts 8. Number of I/O
9. Number of Trials 10. Number of Data Items
11. Data Size 12. Number of Requirements
13. Number of Tasks 14. Number of User Operations
15. Number of System Operations

It can also be observed that, in Table 1, a number of the quantities have a label
starting with ‘number of’. However, these do not use a reference scale typical of
measures in the sciences or in engineering, but are rather counts on entities. For any
of these proposed counts, such as the ‘number of functions’, no specific method is
proposed for an identification of the number of functions in a consistent manner
across measurers and organizations; for instance, the definition of the word ‘function’
could differ from one individual to another within the same organization, and more so
across organizations. Therefore, to say in [10] that such numbers are obtained by an
‘objective’ method is an overstatement, since they must be obtained mostly on the
basis of the judgment of the person carrying out the count.

Of the 15 proposed general quality measure elements, only ‘time’ comes from a
classic base measure using, for instance, the international standard unit of the
second (or a multiple or submultiple of it) as its reference scale. There are also
measuring instruments to ensure that time measurements are indeed obtained in an
objective manner.

It can also be observed that, of the 15 measures proposed in Table 1, at most four
are directly related to the quality of software: number of faults, number of failures,
number of restarts and number of trials. None of the other 12 measures is directly or
indirectly related to the quality of software. In fact, they are strictly independent of it
per se,, as they are often used for normalization purposes, for instance.

Finally, in [14], the issue of documenting a base measure using the full set of
metrology concepts about quantities and units has been investigated and
recommendations provided that would lead to a more comprehensive design of the
software measure.

2.4 Lack of reference to corresponding ISO measurement standards

For the ‘product size’ general quality measure element, [10] lists many ways to
measure product size: lines of code, function points, modules, classes and visual
structures. There are also various methods for counting lines of code and for
measuring function points. Therefore, this general quality measure element could be
further split into different base measures. Moreover, the ISO has specified mandatory
requirements for function point measurement methods [12], and has recognized four
different functional size measurement methods as ISO standards meeting these
requirements, such as COSMIC-FFP [13]. None of these existing ISO software

Position Papers 41
engineering standards, which are referenced in ISO 90003 [15], has been mentioned
or referenced in [10]. Also, the various methods available to obtain those numbers
have their strengths and weaknesses, from a measurement perspective, in terms of
repeatability, reproducibility, software domains of applicability and accuracy.

3 Coverage Harmonization

3.1 Limited coverage of the ISO quality models and corresponding measures

ISO TR 9126, parts 2 to 4, presents the ISO inventory of measures for the full
coverage of the ISO software product quality models (internal quality, external quality
and quality in use) for measuring any of their quality characteristics and
subcharacteristics. The full sets of base measures in these three parts of ISO 9126
are presented in Appendix A and include 82 base measures.

Of these 82 base measures, only 15 are included in [10]; this means that the
coverage in [10] is very limited, and the reasons for this are not obvious. The
proposed content coverage of this subset of base measures is claimed in [10] to be
the ‘most important’; however, no specific criteria to determine its ‘importance’ are
provided. Some generic information is provided in [10] to suggest that these
measures were derived from a questionnaire-based survey; however, it does not
provide the reader with information about the criteria for selection, the size and
representativeness of the sample in the countries where the data were collected, or
the representativeness of this sample outside these countries. Another claim, that
“they represent a default kernel of quality measures, which are proven to be
beneficial and common practice” [10], is not supported by documented evidence, nor
is there a discussion of its generalizability outside its data collection context.

Appendix B presents a detailed analysis of the coverage of the quality measures in
[10], together with the corresponding availability in ISO TR 9126, parts 2 to 4.
Appendix B specifically illustrates that 15 measures for the ‘internal quality’ of
software product are selected in [10] out of an inventory of 70 in the corresponding
ISO TR 9126-3, while 55 measures are excluded, again without a documented
rationale.

Furthermore, the 15 measures of internal quality selected in [10] cover only 4 of the 6
quality characteristics of the ISO model of internal quality, and only 9 of 27
subcharacteristics; again, the rationale for excluding any characteristic or
subcharacteristic is not documented.

Similarly, for the ‘Quality in Use’ quality measures, [10]:

- Includes only 2 quality measures of the 15 already available in ISO TR 9126-4
- Excludes 2 QIU characteristics, that is, ‘safety’ and ‘satisfaction’
- Does not include any Specific Quality Measure Elements related to the ‘Number
of User Operations’ and ‘Number of System Operations’

Position Papers 42
3.2 Overlapping issues

Some additional information included in [10] has already been covered in ISO TR
9126 documents, and will be included in the ISO 25000 series; for instance,
information about the ‘scale types’ is covered through rephrasing information
contained in other documents, once again increasing synchronization and
harmonization right away and over the long term. Similarly for the narratives about
the measures of internal software quality, external software quality and software
quality in use, as well as for the narratives about the software measurement
methods.

This is contrary to the ISO practice of avoiding duplication or the rephrasing of
information across ISO documents, and increases the possibility of inconsistencies
across documents; it could later lead to significant effort over the long term in
maintaining synchronization of documents covering similar subsets of information.

These examples point to configuration management issues over the long term which
will represent additional cost to the purchasers of these ISO documents, since they
will be required to pay twice for the same information which is a subset of the full
inventory. This could lead to some confusion for standards users as to which of these
documents is most valuable to a standard purchaser, and under what circumstances.

4 Discussion

4.1 Summary of harmonization issues in ISO DTR 25021

The ISO is now working on the next generation of software product quality standard,
which will be referred to as Software Product Quality Requirements and Evaluation
(ISO 25000). One of the objectives of this new ISO 25000 series (and what
differentiates it from the current ISO 9126 series) is the harmonization of its contents
with the software measurement terminology of ISO 15939 [8], itself based on the
ISO metrology terminology [9]. In this paper, terminology harmonization issues have
been identified, as well as the coverage of harmonization issues in ISO DTR 25021
and ISO 9126 in terms of the coverage of ISO quality models.

Below is a summary of the harmonization issues identified:

A) Terminology in [10]:
- what is referred to as a ‘quality measure element’ corresponds to the classic
concept of ‘base measure’ in ISO 15939;
- what is referred to as ‘software quality measure’:
- corresponds to the classic concept of ‘derived measure’ in ISO 15939;
- is not at the proper level of abstraction for the concept being measured
when mapped to the hierarchy of concepts for software product quality
adopted by the ISO.

B) Harmonization with the Information Model of ISO 15939:

Position Papers 43
- unless the terminology is harmonized with ISO International Vocabulary of Basic
and General Terms in Metroogy, then it is challenging to align the older versions
of the ISO 9126 and ISO 14598, and it will be even more challenging with the
upcoming updates in ISO 25000.
- should the harmonization of terminology be accepted, it becomes then easier to
map each of these ISO 9126 and 14598 series into the Information Model of
ISO 15939.

C) Description harmonization:
- A large number of the base measures proposed are counts of entities rather
than measures per se with required metrological characteristics, such as: unit,
scale, dimension, measurement method, measurement procedures, etc.
- In [10], in some instances, like ‘product size’ for example, there is no reference
to other existing ISO standards for software size, such as ISO 19761, etc.
- There are a number of claims that the proposed base measures are ‘objective’,
while they are obviously derived from a manual process without precisely
documented measurement procedures, thereby leaving much to the measurer’s
judgment.

D) Coverage harmonization in [10]:
- The set of base measures documented represents only a limited subset of the
base measures within ISO 9126, parts 2 to 4; the rationale for inclusion or
exclusion is not documented.
- The set of base measures does not allow coverage of the full spectrum of
quality characteristics and subcharacteristics in ISO 9126, parts 2 to 4; again,
the rationale for inclusion or exclusion is not documented.

4.2 Recommendations

From the above analysis, the following recommendations are put forward:

- Ensure that the terminology on software product quality measurement is fully
aligned with the classic measurement terminology in the sciences and in
engineering;
- Provide full coverage of the base measures for all three ISO models of software
quality;
- Provide improved documentation of the base measure using the criteria from
metrology;
- Provide clear mapping and traceability of the new ISO 25000 documents to the
ISO 15939 Information Model.

Acknowledgments
The opinions expressed in this paper are solely those of the authors.

Position Papers 44
References
[1] ISO/IEC, ISO/IEC IS 9126, Software Product Evaluation - Quality Characteristics and
Guidelines for Their Use, Geneva, Switzerland: International Organization for
Standardization, 1991.
[2] ISO/IEC, ISO/IEC 9126-1: Software Engineering - Product Quality - Part 1: Quality
Model, Geneva, Switzerland: International Organization for Standardization, 2001.
[3] ISO/IEC, ISO/IEC TR 9126-2: Software Engineering - Product Quality - Part 2: External
Metrics, Geneva, Switzerland: International Organization for Standardization, 2003.
[4] ISO/IEC, ISO/IEC TR 9126-3: Software Engineering - Product Quality - Part 3: Internal
Metrics, Geneva, Switzerland: International Organization for Standardization, 2003.
[5] ISO/IEC, ISO/IEC TR 9126-4: Software Engineering - Product Quality - Part 4: Quality
in Use Metrics, Geneva, Switzerland: International Organization for Standardization,
2004.
[6] M. Azuma, "SQuaRE: The next Generation of ISO/IEC 9126 and 14598 International
Standards Series on Software Product Quality," Proceedings of the European Software
Control and Metrics Conference (ESCOM), London, UK, 2001. pp. 337-346.
[7] ISO/IEC, ISO/IEC 25000: Software Engineering - Software Product Quality
Requirements and Evaluation (SQuaRE) - Guide to SQuaRE, Geneva, Switzerland:
International Organization for Standardization, 2005.
[8] ISO/IEC, ISO/IEC IS 15939: Software Engineering - Software Measurement Process,
Geneva, Switzerland: International Organization for Standardization, 2002.
[9] ISO/IEC, International Vocabulary of Basic and General Terms in Metrology (VIM),
Geneva, Switzerland: International Organization for Standardization, 1993.
[10] SC7, ISO/IEC DTR 25021: Software Engineering - Software Product Quality
Requirements and Evaluation (SQuaRE) - Quality Measure Elements ISO/IEC
JTC1/SC7 WG6, January 11, 2006, 6N-565, 2006.
[11] A. Abran, R. E. Al-Qutaish, J. M. Desharnais, and N. Habra, "An Information Model for
Software Quality Measurement with ISO Standards," in Proceedings of the
International Conference on Software Development (SWDC-REK), Reykjavik, Iceland,
2005. pp. 104-116.
[12] ISO/IEC, ISO/IEC 14143-1: Software Measurement - Functional Size Measurement -
Part 1: Definition of Concepts, Geneva, Switzerland: International Organization for
Standardization, 1998.
[13] ISO/IEC, ISO/IEC 19761: Software Engineering - COSMIC-FFP - A Functional Size
Measurement Method, Geneva, Switzerland: International Organization for
Standardization, 2003.
[14] A. Abran, R. Al Qutaish, and J. Cuadrado, "Investigation of the Metrology Concepts
within ISO 9126 on Software Product Quality Evaluation," submitted to the fourth
Workshop on Software Quality - 28th International Conference on Software
Engineering - ICSE 2006, Shanghai, China, July 2006.
[15] ISO/IEC, ISO/IEC 90003: Software Engineering - Guidelines for the Application of ISO
9001:2000 to Computer Software, Geneva, Switzerland: International Organization for
Standardization, 2004.

The following apendices were not included in the journal due to space constraints.
Appendix A

List of ISO 9126 Base Measures
External Quality - Base Measures
1 Number of Functions
2 Operation Time
3 Number of Inaccurate Computations Encountered by Users
4 Total Number of Data Formats
5 Number of Illegal Operations
6 Number of Items Requiring Compliance
7 Number of Interfaces Requiring Compliance
8 Number of Faults
9 Number of Failures
10 Product Size
11 Number of Test Cases
12 Number of Breakdowns
13 Time to Repair
14 Down Time
15 Number of Restarts
16 Number of Restoration Required
17 Number of Tutorials
18 Number of I/O Data Items
19 Ease of Function Learning
20 Number of Tasks
21 Help Frequency
22 Error Correction
23 Number of Screens or Forms
24 Number of User Errors or Changes
25 Number of Attempts to Customize
26 Total Number of Usability Compliance Items Specified
27 Response Time
28 Number of Evaluations
29 Turnaround Time
30 Task Time
31 Number of I/O Related Errors
32 User Waiting Time of I/O Device Utilization
33 Number of Memory Related Errors
34 Number of Transmission Related Errors
35 Transmission Capacity
36 Number of Revised Versions
37 Number of Resolved Failures
38 Porting User Friendliness
Internal Quality - Base Measures
1 Number of Functions
2 Number of Data Items
3 Number of Data Formats
4 Number of Interface Protocols

5 Number of Access Types
6 Number of Access Controllability Requirements
7 Number of Instances of Data Corruption
8 Number of Compliance Items
9 Number of Interface Requiring Compliance
10 Number of Faults
11 Number of Test Cases
12 Number of Restoration
13 Number of Input Items Which Could Check for Valid Data
14 Number of Operations
15 Number of Messages Implemented
16 Number of Interface Elements
17 Response Time
18 Turnaround Time
19 I/O Utilization (Number of Buffers)
20 Memory Utilization
21 Number of Lines of Code Directly Related to System Calls
22 Number of I/O Related Errors
23 Number of Memory Related Errors
24 Number of Items Required to be Logged
25 Number of Modifications Made
26 Number of Variables
27 Number of Diagnostic Functions Required
28 Number of Entities
29 Number of Built-in Test Function Required
30 Number of Test Dependencies on Other System
31 Number of Diagnostic Checkpoints
32 Number of Data Structures
33 Total Number of Setup Operations
34 Number of Installation Steps
Quality in use - Base Measures
1 Task Effectiveness
2 Total Number of Tasks
3 Task Time
4 Cost of the Task
5 Help Time
6 Error Time
7 Search Time
8 Number of Users
9 Total Number of People Potentially Affected by the System
10 Total Number of Usage Situations

Appendix B

Table B1: External Quality – ISO TR 9126-2 and ISO DTR 25021
Quality
Characteristics
Quality
Subcharacteristics Measure Names
ISO DTR
25021
ISO
9126-2
1 Computational accuracy √ √
2 Precision √ √
Accuracy

3 Accuracy relative to expectations √
4 Data exchangeability (Data format-based) √ √ Interoperability
5 Data exchangeability (User’s success, attempt-based) √
6 Access controllability √ √
7 Access auditability √
Security
8 Data corruption prevention √
9 Funcional implementation completeness √ √
10 Functional adequacy √ √
11 Functional implementation coverage √ √
Suitability


12 Functional specification stability (volatility) √
13 Functional compliance √
Functionality






Functionality
Compliance 14 Interface standard compliance √
15 Failure density against test cases √ √
16 Failure resolution √ √
17 Fault removal √ √
18 Mean time between failures (MTBF) √ √
19 Test maturity √ √
20 Estimated latent fault density √ √
21 Fault density √ √
Maturity




22 Test coverage (Specified operation scenario testing coverage) √
23 Restartability √ √
24 Availability √
25 Mean down time √
26 Mean recovery time √
27 Restorability √
Recoverability
28 Restore effectiveness √
29 Breakdown avoidance √
30 Failure avoidance √
Fault Tolerance
31 Incorrect operation avoidance √
Reliability





Reliability
Compliance 32
Reliability compliance √
33 Effectiveness of the user documentation and/or help system √ √
34 Help accessibility √ √
35 Ease of function learning √
36 Ease of learning to perform a task in use √
37 Effectiveness of user documentation and/or help system in use √
Learnability

38 Help frequency √
39 Physical accessibility √ √
Usability





Operability
40 Operational consistency in use √

Quality
Characteristics
Quality
Subcharacteristics Measure Names
ISO DTR
25021
ISO
9126-2
41 Error correction √
42 Error correction in use √
43 Default value availability in use √
44 Message understandability in use √
45 Self-explanatory error messages √
46 Operational error recoverability in use √
47 Time between human error operations in use √
48 Undoability (User error correction) √
49 Customizability √
50 Operation procedure reduction √
51 Completeness of description √ √
52 Function understandability √ √
53 Understandable input and output √ √
54 Demonstration accessibility √
55 Demonstration accessibility in use √
56 Demonstration effectiveness √
Understandability


57 Evident functions √
58 Attractive interaction √ Attractiveness
59 Interface appearance customizability √
Usability
Compliance 60
Usability compliance √
61 I/O loading limits √ √
62 Maximum memory utilization √ √
63 Maximum transmission utilization √ √
64 Mean occurrence of transmission error √ √
65 I/O device utilization √
66 I/O-related errors √
67 Mean I/O fulfillment ratio √
68 User waiting time of I/O device utilization √
69 Mean occurrence of memory errors √
70 Ratio of memory error/time √
71 Media device utilization balancing √
72 Mean transmission error per time √
Resource
Utilization



73 Transmission capacity utilization √
74 Response time (Mean time to respond) √ √
75 Throughput (Mean amount of throughput) √ √
76 Turnaround time (Mean time for turnaround) √ √
77 Response time √
78 Response time (Worst case response time ratio) √
79 Throughput √
80 Throughput (Worst case throughput time ratio) √
81 Turnaround time √
82 Turnaround time (Worst case turnaround time ratio) √
Efficiency






Time Behavior


83 Waiting time √

Quality
Characteristics
Quality
Subcharacteristics Measure Names
ISO DTR
25021
ISO
9126-2
Efficiency
Compliance 84
Efficiency compliance √
85 Audit trail capability √ √
86 Diagnostic function support √
87 Failure analysis capability √
88 Failure analysis efficiency √
Analyzability
89 Status monitoring capability √
90 Software change control capability √ √
91 Change cycle efficiency √
92 Change implementation elapsed time √
93 Modification complexity √
Changeability
94 Parameterized modifiability √
95 Change success ratio √ Stability
96 Modification impact localization (Emerging failure after change) √
97 Availability of built-in test function √
98 Retest efficiency √
Testability
99 Test restartability √
Maintainability

Maintainability
Compliance 100
Maintainability compliance √
101 Adaptability of data structures √ √
102
Hardware environmental adaptability
(adaptability to hardware devices and
network facilities)
√ √
103
System software environmental
adaptability (adaptability to OS, network
software and cooperated application
software)
√ √
104
Organizational environment adaptability
(Organization adaptability to
infrastructure of organization)
√
Adaptability


105 Porting user-friendliness √
106 Ease of installation √ √ Installability
107 Ease of setup retry √
Coexistance
Replaceability 108
Availability coexistence √
109 Continued use of data √
110 Function inclusiveness √
Replaceability
111 User support functional consistency √
Portability



Portability
Compliance 112
Portability Compliance √

Table B2: Internal Quality – ISO TR 9126-3 and ISO DTR 25021
Quality
Characteristics
Quality
Subcharacteristics Measure Names
ISO DTR
25021
ISO
9126-3
1 Computational accuracy √ √ Accuracy
2 Precision √ √
3 Data exchangeability (Data format-based) √ √ Interoperability
4 Interface consistency (protocol) √
5 Access controllability √ √
6 Access auditability √
7 Data corruption prevention √
Security
8 Data encryption √
9 Funcional implementation completeness √ √
10 Functional adequacy √ √
11 Functional implementation coverage √ √
Suitability


12 Functional specification stability (volatility) √
13 Functional compliance √
Functionality






Functionality
Compliance 14 Intersystem standard compliance √
15 Fault removal √ √
16 Fault detection √
Maturity
17 Test adequacy √
18 Restorability √ Recoverability
19 Restoration effictiveness √
20 Failure avoidance √ Fault Tolerance
21 Incorret operation avoidance √
Reliability
Reliability
Compliance 22
Reliability Compliance √
Learnability 23 Completeness of user documentation and/or help facility √ √
24 Physical accessibility √ √
25 Input validity checking √
26 User operation cancellability √
27 User operation undoability √
28 Customizability √
29 Operation status monitoring capability √
30 Operational consistency √
31 Message clarity √
32 Interface element clarity √
Operability
33 Operational error recoverability √
34 Completeness of description √ √
35 Function understandability √ √
36 Demonstration capability √
Understandability

37 Evident functions √
38 Attractive interaction √ Attractiveness
39 User interface appearance customizability √
Usability



Usability
Compliance 40
Usability Compliance √
41 I/O utilization √
42 I/O utilization message density √
43 Memory utilization √
Efficiency Resource
Utilization
44 Memory utilization message density √

Quality
Characteristics
Quality
Subcharacteristics Measure Names
ISO DTR
25021
ISO
9126-3
45 Transmission utilization √
46 Response time √
47 Throughput time √
Time Behavior
48 Turnaround time √
Efficiency
Compliance 49
Efficiency compliance √
50 Activity recording √ Analyzability
51 Readiness of diagnostic function √
Changeability 52 Change recordability √
53 Change impact √ Stability
54 Modification impact localization √
55 Completeness of built-in test function √
56 Autonomy of testability √
Testability
57 Test progress observability √
Maintainability
Maintainability
Compliance 58
Maintainability compliance √
59 Adaptability of data structures √ √
60
Hardware environmental adaptability
(adaptability to hardware devices and
network facilities)
√ √
61
System software environmental
adaptability (adaptability to OS, network
software and cooperated application
software)
√ √
62 Organizational environment adaptability √
Adaptability


63 Porting user-friendliness √
64 Ease of setup retry √
65 Installation effort √
Installability
66 Installation flexibility √
Co-existence 67 Availability of coexistence √
68 Continued use of data √ Replaceability
69 Functional inclusiveness √
Portability


Portability
Compliance 70
Portability compliance √

Table B3: Quality in Use – ISO TR 9126-4 and ISO DTR 25021
Quality Characteristics Measure Names ISO DTR 25021
ISO
9126-4
1 Task effectiveness √
2 Task completion √ √
Effectiveness

3 Error frequency √
4 Task time √ √
5 Task efficiency √
6 Economic productivity √
7 Productive proportion √
Productivity
8 Relative user efficiency √
9 User health and safety √
10 Safety of people affected by use of the system √
11 Economic damage √
Safety


12 Software damage √
13 Satisfaction scale √
14 Satisfaction questionnaire √
Satisfaction
15 Discretionary usage √