The State of the Art in Automating Usability Evaluation
of User Interfaces
MELODY Y. IVORY AND MARTI A. HEARST
University of California, Berkeley
Usability evaluation is an increasingly important part of the user interface design
process. However, usability evaluation can be expensive in terms of time and human
resources, and automation is therefore a promising way to augment existing
approaches. This article presents an extensive survey of usability evaluation methods,
organized according to a new taxonomy that emphasizes the role of automation. The
survey analyzes existing techniques, identifies which aspects of usability evaluation
automation are likely to be of use in future research, and suggests new ways to expand
existing approaches to better support usability evaluation.
Categories and Subject Descriptors: H.1.2 [Information Systems]: User/Machine
Systems—human factors; human information processing; H.5.2 [Information
Systems]: User Interfaces—benchmarking; evaluation/methodology; graphical user
interfaces (GUI )
General Terms: Human Factors
Additional Key Words and Phrases: Graphical user interfaces, taxonomy, usability
evaluation automation, web interfaces
1. INTRODUCTION
Usability is the extent to which a computer
system enables users, in a given context
of use, to achieve specified goals effec-
tively and efficiently while promoting feel-
ings of satisfaction.
1
Usability evaluation
(UE) consists of methodologies for mea-
suring the usability aspects of a system’s
user interface (UI) and identifying specific
problems [Dix et al. 1998; Nielsen 1993].
1
Adapted from ISO9241 [International Standards
Organization 1999].
This research was sponsored in part by the Lucent Technologies Cooperative Research Fellowship Program,
a GAANN fellowship, and Kaiser Permanente.
Authors’ addresses: M. Y. Ivory, Computer Science Division, University of California, Berkeley, Berkeley,
CA 94720-1776; email: ivory@CS.Berkeley.edu; M. A. Hearst, School of Information Management and
Systems, University of California, Berkeley, Berkeley, CA 94720-4600; email: hearst@SIMS.Berkeley.edu.
Permission to make digital/hard copy of part or all of this work for personal or classroom use is granted
without fee provided that the copies are not made or distributed for profit or commercial advantage, the
copyright notice, the title of the publication, and its date appear, and notice is given that copying is by
permission of the ACM, Inc. To copy otherwise, to republish, to post on servers, or to redistribute to lists,
requires prior specific permission and/or a fee.
c©2001 ACM 0360-0300/01/1200–0470 $5.00
Usability evaluation is an important part
of the overall user interface design pro-
cess, which consists of iterative cycles
of designing, prototyping, and evaluating
[Dix et al. 1998; Nielsen 1993]. Usability
evaluation is itself a process that entails
many activities depending on the method
employed. Common activities include.
—Capture collecting usability data, such
as task completion time, errors, guide-
line violations, and subjective ratings;
—Analysis interpreting usability data to
identify usability problems in the inter-
face; and
ACM Computing Surveys, Vol. 33, No. 4, December 2001, pp. 470–516.

Automating Usability Evaluation of User Interfaces 471
—Critique: suggesting solutions or im-
provements to mitigate problems.
A wide range of usability evaluation
techniques have been proposed, and a sub-
set of these is currently in common use.
Some evaluation techniques, such as for-
mal user testing, can only be applied af-
ter the interface design or prototype has
been implemented. Others, such as heuris-
tic evaluation, can be applied in the early
stages of design. Each technique has its
own requirements, and generally differ-
ent techniques uncover different usability
problems.
Usability findings can vary widely when
different evaluators study the same user
interface, even if they use the same eval-
uation technique [Jeffries et al. 1991;
Molich et al. 1998, 1999; Nielsen 1993].
Two studies in particular, the first and
second comparative user testing studies
(CUE-1 [Molich et al. 1998] and CUE-2
[Molich et al. 1999]), demonstrated less
than a 1% overlap in findings among four
and eight independent usability testing
teams for evaluations of two user inter-
faces. This result implies a lack of sys-
tematicity or predictability in the findings
of usability evaluations. Furthermore, us-
ability evaluation typically only covers a
subset of the possible actions users might
take. For these reasons, usability experts
often recommend using several different
evaluation techniques [Dix et al. 1998;
Nielsen 1993].
How can systematicity of results and
fuller coverage in usability assessment be
achieved? One solution is to increase the
number of usability teams evaluating the
system and to increase the number of
study participants. An alternative is to au-
tomate some aspects of usability evalua-
tion, such as the capture, analysis, or cri-
tique activities.
Automation of usability evaluation has
several potential advantages over nonau-
tomated evaluation, such as the following.
—Reducing the cost of usability evalua-
tion. Methods that automate capture,
analysis, or critique activities can de-
crease the time spent on usability eval-
uation and consequently the cost. For
example, software tools that automati-
cally log events during usability testing
eliminate the need for manual logging,
which can typically take up a substan-
tial portion of evaluation time.
—Increasing consistency of the errors
uncovered. In some cases it is possible
to develop models of task completion
within an interface, and software tools
can consistently detect deviations from
these models. It is also possible to
detect usage patterns that suggest
possible errors, such as immediate task
cancellation.
—Predicting time and error costs across
an entire design. As previously dis-
cussed, it is not always possible to
assess every single aspect of an inter-
face using nonautomated evaluation.
Software tools, such as analytical
models, make it possible to widen the
coverage of evaluated features.
—Reducing the need for evaluation ex-
pertise among individual evaluators.
Automating some aspects of evaluation,
such as the analysis or critique activi-
ties, could aid designers who do not have
expertise in those aspects of evaluation.
—Increasing the coverage of evaluated
features. Due to time, cost, and resource
constraints, it is not always possible
to assess every single aspect of an
interface. Software tools that generate
plausible usage traces make it possible
to evaluate aspects of interfaces that
may not otherwise be assessed.
—Enabling comparisons between alter-
native designs. Because of time, cost,
and resource constraints, usability
evaluations typically assess only one
design or a small subset of features
from multiple designs. Some auto-
mated analysis approaches, such as
analytical modeling and simulation,
enable designers to compare predicted
performance for alternative designs.
—Incorporating evaluation within the
design phase of UI development, as
opposed to being applied after imple-
mentation. This is important because
evaluation with most nonautomated
ACM Computing Surveys, Vol. 33, No. 4, December 2001.