Extracting Usability Information from User Interface Events
DAVID M. HILBERT AND DAVID F. REDMILES
University of California at Irvine
Modern window-based user interface systems generate user interface events as natural
products of their normal operation. Because such events can be automatically captured
and because they indicate user behavior with respect to an application’s user interface,
they have long been regarded as a potentially fruitful source of information regarding
application usage and usability. However, because user interface events are typically
voluminos and rich in detail, automated support is generally required to extract
information at a level of abstraction that is useful to investigators interested in
analyzing application usage or evaluating usability.
This survey examines computer-aided techniques used by HCI practitioners and
researchers to extract usability-related information from user interface events. A
framework is presented to help HCI practitioners and researchers categorize and
compare the approaches that have been, or might fruitfully be, applied to this problem.
Because many of the techniques in the research literature have not been evaluated in
practice, this survey provides a conceptual evaluation to help identify some of the
relative merits and drawbacks of the various classes of approaches. Ideas for future
research in this area are also presented.
This survey addresses the following questions: How might user interface events be
used in evaluating usability? How are user interface events related to other forms of
usability data? What are the key challenges faced by investigators wishing to exploit
this data? What approaches have been brought to bear on this problem and how do they
compare to one another? What are some of the important open research questions in
this area?
Categories and Subject Descriptors: H.5.2 [Information Interfaces and
Presentation]: User Interfaces—Evaluation/methodology
General Terms: Human factors, Measurement, Experimentation
Additional Key Words and Phrases: usability testing, user interface event monitoring,
sequential data analysis, human-computer interaction
1. INTRODUCTION
User interface events (UI events) are gen-
erated as natural products of the normal
operation of window-based user interface
systems such as those provided by the
Authors’ address: Department of Information and Computer Science, University of California, Irvine, CA
e-mail: {dhilbert,redmiles}@ics.uci.edu.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted
without fee provided that copies are not made or distributed for profit or direct commercial advantage and
that copies show this notice on the first page or initial screen of a display along with the full citation.
Copyrights for components of this work owned by others than ACM must be honored. Abstracting with
credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists, or to use any
component of this work in other works, requires prior specific permission and/or a fee. Permissions may
be requested from Publications Dept., ACM Inc., 1515 Broadway, New York, NY 10036, USA, fax +1 (212)
869-0481, or permissions@acm.org.
c©2001 ACM 0360-0300/01/1200-0384 $5.00
Macintosh Operating System [Lewis
and Stone 1999], Microsoft Windows
[Petzold 1998], the X Window System
[Nye and O’Reilly 1992], and the Java
Abstract Window Toolkit [Zukowski and
Loukides 1997]. Such events indicate user
ACM Computing Surveys, Vol. 32, No. 4, December 2000, pp. 384–421.

Extracting Usability Information 385
behavior with respect to the components
that make up an application’s user inter-
face (e.g., mouse movements with respect
to application windows, keyboard presses
with respect to application input fields,
mouse clicks with respect to application
buttons, menus, and lists). Because such
events can be automatically captured and
because they indicate user behavior with
respect to an application’s user interface,
they have long been regarded as a po-
tentially fruitful source of information
regarding application usage and usability.
However, because user interface events
are typically extremely voluminous and
rich in detail, automated support is
generally required to extract information
at a level of abstraction that is useful
to investigators interested in analyzing
application usage or evaluating usability.
While a number of potentially related
techniques have been applied to the prob-
lem of analyzing sequential data in other
domains, this paper primarily focuses on
techniques that have been applied within
the domain of HCI. Providing an in-depth
treatment of all potentially related tech-
niques would necessarily limit the amount
of attention paid to characterizing the
approaches that have in fact been brought
to bear on the specific problems associ-
ated with analyzing HCI events. However,
this survey attempts to characterize UI
events and analysis techniques in such a
way as to make comparison between tech-
niques used in HCI and those used in other
domains straightforward.
1.1 Goals and Method
The fundamental goal of this survey is to
construct a framework to help HCI prac-
titioners and researchers categorize, com-
pare, and evaluate the relative strengths
and limitations of approaches that have
been, or might fruitfully be, applied to this
problem. Because exhaustive coverage of
all existing and potential approaches is
impossible, we attempt to identify key
characteristics of existing approaches that
divide them into more or less natural cate-
gories. This allows classes of systems, not
just instances, to be compared. The hope
is that an illuminating comparison can be
conducted at the class level and that clas-
sification of new instances into existing
classes will prove to be unproblematic.
In preparing this survey, we searched
the literature in both academic and
professional computing forums for papers
describing computer-aided techniques for
extracting usability-related information
from user interface events. We selected
and analyzed an initial set of papers to
identify key characteristics that distin-
guish the approaches applied by various
investigators.
We then constructed a two-dimensional
matrix with instances of existing ap-
proaches listed along one axis and char-
acteristics listed along the other. This led
to an initial classification of approaches
based on clusters of related attributes.
We then iteratively refined the compar-
ison attributes and classification scheme
based on further exploration of the litera-
ture. The resulting matrix indicates areas
in which further research is needed and
suggests synergistic combinations of cur-
rently isolated capabilities.
Ideally, an empirical evaluation of these
approaches in practice would help eluci-
date more precisely the specific types of us-
ability questions for which each approach
is best suited. However, because many of
the approaches have never been realized
beyond the research prototype stage, lit-
tle empirical work has been performed to
evaluate their relative strengths and lim-
itations. This survey attempts to provide
a conceptual evaluation by distinguish-
ing classes of approaches and illuminat-
ing their underlying nature. As a result,
this survey should be regarded as a guide
to understanding the research literature
and not as a guide to selecting an already
implemented approach for use in practice.
1.2 Comparison Framework
This subsection introduces the high level
categories that have emerged as a result
of the survey. We present the framework
in more detail in Section 4.
r
Techniques for synchronization and
searching. These techniques allow user
ACM Computing Surveys, Vol. 32, No. 4, December 2000.