Scenario-Based Assessment
of Nonfunctional Requirements
Andreas Gregoriades and Alistair Sutcliffe, Member, IEEE
Abstract—This paper describes a method and a tool for validating nonfunctional requirements in complex socio-technical systems.
The System Requirements Analyzer (SRA) tool validates system reliability and operational performance requirements using scenario-
based testing. Scenarios are transformed into sequences of task steps and the reliability of human agents performing tasks with
computerized technology is assessed using Bayesian Belief Network (BN) models. The tool tests system performance within an
envelope of environmental variations and reports the number of tests that pass a benchmark threshold. The tool diagnoses
problematic areas in scenarios representing pathways through system models, assists in the identification of their causes, and
supports comparison of alternative requirements specifications and system designs. It is suitable for testing socio-technical systems
where operational scenarios are sequential and deterministic, in domains where designs are incrementally modified so set up costs of
the BNs can be defrayed over multiple tests.
Index Terms—Nonfunctional requirements validation, scenario-based testing, Bayesian belief networks, systems engineering.


1 INTRODUCTION
SCENARIOS have attracted considerable interest as a meansof validating requirements specifications [4], [5], [9],
[54]. Foundations of scenario-based approaches were laid
by Davis and Hsia [8] and Hsia et al. [33] and the influential
work of Potts et al. [48] who created the Inquiry Cycle and,
later, the ScenIC [46] method for scenario-based require-
ments validation [46], [47], [48], [49]. The potential of
scenario-based requirements validation has also been
recognised by Anderson and Durley [1], Zhu and Jin [71],
and Haumer et al. [28].
Scenarios have been applied to the analysis of nonfunc-
tional requirements (NFRs) using dependency tables to
assess the relationships between different NFRs [43] and by
modeling the dependencies between goals (representing
functional requirements and nonfunctional requirements,
also called soft goals) and the agents and tasks that achieve
them in the i* language [70]. The “satisfying” or fulfillment
of soft goals (i.e., NFRs) by functional requirements is
assessed by inspecting strategic dependency and rationale
models that show goals, agents, tasks, and dependency
relationships [40], [41], [70]. Although i* support tools do
provide limited reasoning support for assessing dependen-
cies, most validation still requires human expertise. The
TROPOS [20] language supports more formal reasoning
about i* models; however, it does not explicitly assess
nonfunctional requirements.
Unlike functional requirements, which can be determi-
nistically validated, NFRs are soft variables that cannot be
implemented directly; instead, they are satisfied [40] by a
combination of functional requirements. Since many NFRs
are influenced by human properties, they inherit the diverse
nature of human characteristics; for example, assessment of
NFRs such as system reliability is influenced by human
characteristics such as ability, stress, concentration, etc.
Software engineering and systems engineering require-
ments validation methods do not take human factors into
account, even though they are a critical cause of systems
failure [30], [31], [52].
In our previous work [61], we developed a method and
software tool for scenario-based requirements validation
that prompted designers with questions about potential
problems in a scenario event sequence. The tool used a
psychology-based taxonomy of failure causes [32] with a
pathway expansion algorithm that generated alternative
paths from a single seed scenario. It supported an
inspection-based process with probe questions about
possible problems and generic requirements as cures for
the problems it identified. However, evaluation of this
approach showed that too many scenario variations were
generated and the software developers drowned in ex-
cessive detail.
To address this problem, we developed a semiautomated
approach to requirements validation [59] by transforming
the taxonomy of human and system failure causes into a
model to predict potential errors in a system design.
Bayesian Belief Nets (BNs) provided a probabilistic reason-
ing mechanism to predict reliabilities from models com-
posed of descriptions of system components and attributes
of human operators [21]. However, the output from the
BN model was fed into a paper-based walkthrough for
validating scenarios, which was still time-consuming. This
led to the motivation for the research we report in this
392 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 31, NO. 5, MAY 2005
. A. Gregoriades is with the Surrey Defence Technology Centre, School of
Management, University of Surrey, Guildford, Surrey GU2 7XH.
E-mail: A.Gregoriades@surrey.ac.uk.
. A. Sutcliffe is with the School of Informatics, University of Manchester,
PO Box 88, Manchester M60 1QD UK.
E-mail: ags@man.ac.uk.
Manuscript received 12 Oct. 2004; revised 18 Feb. 2005; accepted 11 Mar.
2005; published online 26 May 2005.
Recommended for acceptance by J. Kramer.
For information on obtaining reprints of this article, please send e-mail to:
tse@computer.org, and reference IEEECS Log Number TSE-0208-1004.
0098-5589/05/$20.00
2005 IEEE Published by the IEEE Computer Society