A FRAMEWORK FOR EVALUATING USABILITY
OF CLINICAL MONITORING TECHNOLOGY
Jeremy Daniels, BASc1, Sidney Fels, PhD2, Andre
Kushniruk, PhD3, Joanne Lim, MASc1 and
J. Mark Ansermino, MBBCh, MSc1
Daniels J, Fels S, Kushniruk A, Lim J, Ansermino JM. A framework for
evaluating usability of clinical monitoring technology.
J Clin Monit Comput 2007; 21:323–330
ABSTRACT. Technology design is a complex task, and
acceptability is enhanced when usability is central to its
design. Evaluating usability is a challenge for purchasers and
developers of technology. We have developed a framework for
testing the usability of clinical monitoring technology through
literature review and experience designing clinical monitors.
The framework can help designers meet key international
usability norms. The framework includes these direct testing
methods: thinking aloud, question asking, co-discovery,
performance and psychophysiological measurement. Indirect
testing methods include: questionnaires and interviews,
observation and ethnographic studies, and self-reporting logs.
Inspection, a third usability testingmethod, is also included. The
use of these methods is described and practical examples of how
they would be used in the development of an innovative
monitor are given throughout. This framework is built on a
range of methods to ensure harmony between users and new
clinical monitoring technology, and have been selected to be
practical to use.
KEY WORDS. usability testing, direct testing, indirect testing,
inspection, think aloud, psychophysiology, situation awareness.
INTRODUCTION
Successful technology is socially acceptable, practical,
useful and usable. Technology design is a multifaceted
effort consisting of many important activities. The design
group must consider the technology user’s working
characteristics and current (as well as future) tasks, which
can be accomplished via a task analysis [1]. Additionally,
the design group must understand the user’s environ-
ment, and assess and mitigate the risks associated with
introducing their new technology into an established
work flow. An essential aspect of any technology is that
it is used appropriately, and the design effort should
likewise make efforts to ensure correct usage. However,
before technology can be put to use, it must be accepted
by its users.
Usability is key in the acceptance of a technology by
its users and usability testing is regarded as a major
technique to be used by developers of medical devices
seeking to comply with the American National Standards
Institute’s Human Factors Design Processes for Medical
Devices [2], or the European Usability Collateral Standard,
IEC 60601-1-6 [3]. The purpose of this article is to give
a brief description of usability and to describe a usability
From the 1Department of Anesthesiology, Pharmacology, and
Therapeutics, The University of British Columbia, Room E414A,
4480 Oak Street, Vancouver, BC, Canada V6H 3V4; 2Department
of Electrical and Computer Engineering University of British
Columbia, Vancouver, Canada; 3School of Health Information
Science, University of Victoria, Victoria, Canada.
Received 17 May 2007. Accepted for publication 18 July 2007.
Address correspondence to J. Daniels, Department of Anesthesia,
British Columbia’s Children’s Hospital, Room E414A, 4480 Oak
Street, Vancouver, BC, Canada V6H 3V4.
E-mail: jdaniels2@cw.bc.ca
Journal of Clinical Monitoring and Computing (2007) 21:323–330
DOI: 10.1007/s10877-007-9091-y
Springer 2007

evaluation framework that can be used by purchasers and
developers of clinical monitoring technology. We have
focused on a practical depiction of usability testing
methods. This article has particularly salient use for the
non-human factors expert in designing and evaluating
new monitoring devices.
Usability is composed of the five attributes and goals
listed in Table 1 [4].
Since the topic of usability engineering was first
studied formally in the 1980s, several methods have been
developed to determine how usable a system is. These
usability characteristics can be used to meet some of the
quality requirements when fulfilling usability norms,
such as the ISO 9216 Software Product Evaluation
Quality Model [5].
THE ISO 9216 QUALITY MODEL
The ISO 9216 Quality Model consists of four parts. Part
1, referred to as the Quality Model, specifies six criteria by
which to judge the quality of systems, including clinical
monitors. The criteria are: portability, functionality, reli-
ability, maintainability, efficiency, and usability [5]. Of
these, this framework is applicable to evaluating efficiency
and usability. Efficiency can be assessed via the perfor-
mance measurement, questionnaire, interview, observa-
tion/ethnography, and self-reporting log techniques. All
techniques in this framework can be used to satisfy the
usability component of ISO 9216, Part 1. Part 2 of the
ISO 9216 Product Quality Model, referred to as External
Metrics, assesses the performance of the computer-based
system that will be used to run the software [6]. Our
framework cannot be used by technology developers to
meet these quality metrics. Part 3 of the ISO 9216 Quality
Model, referred to as Internal Metrics [7], can be satisfied
via applying the inspection methods contained in our
framework. Part 4 of the ISO 9216 Quality Model, re-
ferred to as Quality in Use [8], can be satisfied by applying
the questionnaire, interview, observation/ethnography,
and self-reporting log techniques.
USABILITY TESTING
Usability testing provides direct data on user interactions
with technology and is essential in gaining a picture of
how useful the system will be. Usability testing is exper-
imental in nature and data are collected regarding users’
interactions with the technology under investigation.
Methods developed to collect usability data include
video cameras to record the human–technology interac-
tion, one-way mirrors which observers sit behind, a test
plan for the user to follow, and the system under study.
Investigators may substitute a low-fidelity prototype made
of wood or paper if a mature version of the technology is
not readily available. Studies may feature video capture of
the user and the technology itself, and/or the associated
instruction manual. Verbal data are typically analyzed with
annotation software and the statistical methods familiar to
clinical researchers [9].
Usability testing is used to gather data on testable
hypotheses. However, usability testing of a system in
isolation is often is not useful for answering questions such
as ‘‘How usable is the product?’’ because usability can
only be effectively evaluated relative to alternative systems
or pre-established goals. Valid questions for usability
testing include ‘‘Does changing this aspect of the device
make it more learnable?’’, ‘‘Does changing this aspect of
the device result in fewer errors?’’, or ‘‘Does this monitor
design meet our goal that 90% of experienced anesthesi-
ologists can change the respirator setting from a pressure-
controlled setting to volume-controlled within 30 s with
no more than one error?’’.
A standard format for communicating the results of
usability tests in industry has been developed by the
American National Standards Institute, the ANSI-NCITS
354-2001 Common Industry Format For Usability Test
Reports [10]. The usability testing methods described in
this paper are applicable to clinical monitoring technology
at any stage of development: from initial concept brain-
storming, to finalizing the design of a lengthy develop-
ment process. Those wishing to include these techniques
in their technology development efforts would apply the
methods with low-fidelity mock-ups in the early stages of
design, and progress to higher fidelity prototypes as
development efforts progress. The usability engineering
Table 1. The five attributes of usability (adapted from [4])
The five attributes of usability
Learnability. The system should be easy to learn so that the
user can begin using the tool rapidly
Efficiency. The system should be efficient to use so that
once learned, a user can be highly productive
Memorability. The system should be memorable so that
a casual user can return to the system after not using
it and be able to interact with it successfully
Errors. The system should have a low error rate,
and users should be able to easily recover from any
errors that do occur
Satisfaction. The system should be pleasant to use so
that users are subjectively satisfied with the system
324 Journal of Clinical Monitoring and Computing