A Survey of User-Centered Design Practice

Karel Vredenburg
IBM
Toronto, Canada
karel@ca.ibm.com

Ji-Ye Mao
University of Waterloo
Waterloo, Canada
jmao@mansci2.uwaterloo.ca

Paul W Smith
IBM
Toronto, Canada
pswmith@ca.ibm.com


Tom Carey
University of Waterloo
Waterloo, Canada
tcarey@watserv1.uwaterloo.ca
ABSTRACT
This paper reports the results of a recent survey of user-
centered design (UCD) practitioners. The survey involved
over a hundred respondents who were CHI’2000 attendees
or current UPA members. The paper identifies the most
widely used methods and processes, the key factors that
predict success, and the critical tradeoffs practitioners must
make in applying UCD methods and processes. Results
show that cost-benefit tradeoffs are a key consideration in
the adoption of UCD methods. Measures of UCD
effectiveness are lacking and rarely applied. There is also a
major discrepancy between the commonly cited measures
and the actually applied ones. These results have
implications for the introduction, deployment, and
execution of UCD projects.
Keywords
User-centered design, usability engineering, HCI
professionals, methodology, organizational impact
INTRODUCTION
The objective of this research was to investigate the actual
use of user-centered design (UCD) methods in practice
across the industry. Much has been written in the research
literature about UCD. User-Centered Design had its
origins with the seminal work of Norman and Draper
(1986). Others have further operationalized and optimized
the basic approach (Vredenburg, 1999). However, an
examination of practice at major companies from across
the industry discovered that many of the methods that are
discussed in the literature are not effective or practical for
a variety of reasons (Vredenburg & Butler, 1996). There is
a need for practical UCD guidelines based on the
collective wisdom of the industry-wide community of
UCD practitioners.
This paper intends to provide the UCD community
empirical evidence on what works versus what does not,
and what is practical and what is not.
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
otherwise, or republish, to post on servers or to redistribute to lists,
requires prior specific permission and/or a fee.
CHI 2002, April 20-25, 2002, Minneapolis, Minnesota, USA.
Copyright 2002 ACM 1-58113-453-3/02/0004…$5.00.

It also sheds light on the usefulness and usability of
common UCD methods, which is expected to lead to
practical guidelines and evaluative criteria. Whereas a
recent survey has focused on “strategic usability” in terms
of embedding usability engineering in organizational
processes and culture, and contributing to corporate-wide
decision making and product decisions (Rosenbaum,
Rohn, & Humburg, 2000), our study addresses product
usability itself. More specific research questions include:
Which UCD methods are most widely used and why?
What are the benefits and weaknesses of each method in
the eyes of the practitioners? What are the organizational
impacts of UCD and what measures are in place to assess
progress?
Results of this research were expected to provide an
empirical basis for UCD planning, training, adoption and
execution. For example, future UCD practice could benefit
from knowledge of key success factors for the most widely
used methods and techniques, common difficulties and
concerns with various methods, and cost-benefit tradeoffs.
In addition, it was expected that results could confirm the
importance of UCD as well-established and widely
accepted as an informal survey by Hudson (2000) has
indicated. Our study attempted to further Hudson’s work
by conducting a large scale, carefully designed and
executed survey. The focus of the study was on the
perspectives of individual UCD practitioners in terms of
their personal perception and experiences working within
their respective companies.
LITERATURE REVIEW
Prior studies are generally in agreement that traditional
software development tools and practices have
disappointingly small effects on improving software
development (e.g., Curtis, Krasner, & Iscoe, 1988). Gould
et al. claimed that the user-centered design process was
still not often used in practice due to both organizational
and technical reasons even though the approach had been
in existence for over a decade (1991). Further, Nielsen
(1994) argued that many developers do not use usability
engineering techniques because they are considered
intimidating in their complexity, too time consuming and
expensive.
It is of critical importance to the UCD community to
determine whether the situation has changed over the past
few years. Not coincidentally, several surveys have been
minneapolis, minnesota, usa • 20-25 april 2002 Paper: Design Methods


Volume No, 4, Issue No. 1 471

conducted recently on UCD practice, reflecting the need
within the UCD community for this information.
Rosenbaum et al. (2000) surveyed 134 CHI professionals
with a focus on the contribution of organizational
approaches and UCD methods to strategic usability. It was
found that major obstacles to creating greater strategic
impact include resource constraints, which were mentioned
by 28.6% of the respondents, resistance to user-centered
design or usability, lack of knowledge about usability.
Partnering with marketing was identified as a very
effective approach.
Hudson, along with Bevan, conducted an email-based
informal survey of UCD (Hudson, 2000). Questionnaires
were posted to several mailing lists of HCI groups, and
resulted in 102 responses from mostly usability
practitioners. The most commonly used methods, as
reflected in the percentage of respondents using them,
include informal usability testing, user analysis/profiling,
evaluating existing systems, low-fidelity prototyping,
heuristic evaluation, task identification, navigation design,
scenario-based design.
It appears that informal and less structured methods tend to
be used much more widely than more formal and
structured methods. For example, ranked on top are
informal usability testing, low-fidelity prototyping, and
heuristics, whereas more formal methods are ranked at the
bottom such as focus groups, cognitive walkthrough.
A 10-question web survey was conducted recently
involving 100 usability practitioners (Gunther, Janis, &
Butler, 2001). The most successful activities identified by
the respondents included usability testing, which was
mentioned by 39% of the respondents, prototyping, and
heuristic evaluation, confirming Hudson’s finding (2000).
In addition, they identified the top three best selling
activities across the development lifecycle include
customer interviews, paper or other prototyping, and
usability test. This survey also examined several key
aspects of the organizational context of UCD such as
developer resistance to UCD and the interaction between
the UCD specialist and developers, design team
composition and mission, and successes and failures with
UCD processes.
These prior surveys have produced valuable insights about
UCD practice, and each of them has its own focus and
viewpoint. Our survey has several unique features most
notably the assessment of the overall organizational impact
of UCD and measures of UCD success, the profile of and
processes used in a typical UCD project, a comprehensive
assessment of the strengths and weaknesses of UCD
methods as they are currently practiced in product
development environments. It allowed us to investigate the
typical costs and benefits of carrying out UCD, and
empirically determine the factors most related to the
effectiveness of UCD.

RESEARCH METHOD
A working definition of UCD was given at the beginning
of the questionnaire as follows: “UCD is herein
considered, in a broad sense, the practice of the following
principles, the active involvement of users for a clear
understanding of user and task requirements, iterative
design and evaluation, and a multi-disciplinary approach.
UCD methods are modular or identifiable processes
involved in UCD practice. You should NOT think of UCD
as merely usability testing or software engineering.”
The questionnaire consisted of several general questions
on the overall impact of UCD methods in practice, and
specific questions on a representative UCD project, and
detailed assessment of five commonly used UCD methods
to be identified by respondents based on their own
experience
1
. There were Likert-type scales, multiple choice
and qualitative questions in the free text form. The
questionnaire was extensively pre-tested by members of
IBM’s UCD Advisory Council and by members of the
TeleCHI list. The questionnaire was revised based on the
feedback from these tests.
The target respondents were experienced practitioners of
UCD who had at least three years of experience with UCD,
and considered UCD as their primary job. The invitation
and questionnaire were distributed to CHI’2000 attendees
and Usability Professional Association (UPA) members. In
the invitation, the required qualification was highlighted
and only those who qualified were asked to participate.
The survey was first distributed to CHI’2000 attendees
towards the end of 2000. In early 2001, the UPA
management office emailed our invitation and
questionnaire to their members directly. A week after the
distribution, a reminder was sent to the non-respondents to
encourage response, along with the questionnaire.
The response rate in both cases was about 3%. Since we
have no information on how many of the CHI’2000
attendees and UPA members actually belonged to the
target sample of practitioners with at least three years of
experience, the response rate is different from the typical
response rate reported in other surveys. The real response
rate could be many times higher than 3%. The two samples
were compared and no statistically significant differences
were detected in the quantitative answers. Therefore, they
were combined for data analysis.
We did not ask our respondents to identify their companies
expecting some might not wish to release the information
despite our promise of anonymity. However, judging from
respondents’ email addresses, we know 10 individuals
from three of the largest companies in the IT industry
participated in this study. No other respondents appeared
to be from the same company, but we cannot be certain
about this as some respondents used generic email servers.
Our respondents could be from up to 96 companies of

1
The questionnaire is not attached due to space limitations, but it
can be requested from the authors.
Paper: Design Methods CHI changing the world, changing ourselves


472 Volume No, 4, Issue No. 1