Note: In this on-line PDF version the format for page numbers has been changed, but the
content is the same than in the journal.
THE JOURNAL OF THE LEARNING SCIENCES, 6(3), 271-315
Copyright 1997, Lawrence Erlbaum Associates, Inc.
Quantifying Qualitative Analyses of Verbal Data: A Practical Guide
Michelene T. H. Chi
Learning Research and Development Center
University of Pittsburgh
This article provides one example of a method of analyzing qualitative data in an objective and quantifiable
way. Although the application of the method is illustrated in the context of verbal data such as explanations,
interviews, problem-solving protocols, and retrospective reports, in principle, the mechanics of the method
can be adapted for coding other types of qualitative data such as gestures and videotapes. The mechanics of the
method we outlined in 8 concrete step. Although verbal analyses can be used for many purposes, the main
goal of the analyses discussed here is to formulate an understanding of the representation of the knowledge
used in cognitive performances and how that representation changes with learning This can be contrasted with
another method or analyzing verbal protocols, the goal of which is to validate the cognitive processes of
human performance, often as embodied in a computational model.
For a variety of reasons, there has been an increasing need in cognitive science and educational
research to collect and analyze "messy" data. Messy data refer to such things as verbal explanations,
observations, and videotapings, as well as gestures. One reason for the need to collect this kind of
data is the trend toward studying complex activities in practice or in the context in which they occur.
So, for example, to understand how an apprentice learns a trade, one might have to observe the
learner in context. Likewise, it is becoming increasingly clear that the performance of experts (such
as industrial and software designers) relies on the use of external aids and tools, such as notes and
drawings (Norman, 1988). Thus, to capture a complete understanding of their skill, ideally one should
incorporate in the analysis not only their verbal transcripts but also their drawings, pointings, and
gesturings (Tang, 1989). Of course, both verbal data and observational data have been used widely for
some time, in cognitive simulation research for the former case and in anthropological studies for the
latter. However, it has been discouraging for novice students of cognitive science and education to
adopt these methods for various reasons, such as the restricted applicability of the protocol analysis
method (see Ericsson & Simon, 1984), the subjectiveness of the observational methods (see Schofield
& Anderson, 1987), and the time-consumingness of both of these methods. The goal of this article is
to attempt to provide guidance for how one can approach an analysis of verbal data more generally,
involving a method that integrates elements of qualitative and quantitative analyses so that the
interpretation of the results is less subjective. Unfortunately, the time-consuming nature of
qualitative-based quantitative analysis will remain, even with an explicit guide. (This article does not,
however, address the analysis of videotape data because that topic is already covered by Jordan &
Henderson, 1995, and it does not cover the analysis of gestures, which is discussed in Goldin-Meadow,
Alibali, & Breckinridge Church, 1993).
There are basically two ways to write such a guide. One way is to survey the literature, identify all
those studies that have used some kind of qualitative analysis of verbal data, then describe, analyze,
and synthesize all the various methods. Such a survey would have to include not only studies that use
some kind of qualitative analysis of verbal data (such as those of Patel & Groen, 1986; Ranney,
1994; Trabasso & Sub, 1993; Voss, Tyler, & Yengo, 1983) but also to include research that focuses
explicitly on comparing different methods of collecting and analyzing data, such as the work of
Geiwitz, Klatsky, and McCloskey (1988) and Hoffman (1987). This approach is not taken here for
two reasons. First, an analysis, synthesis, and comparison type of exhaustive review of the literature
focusing on different aspects of verbal analyses has already been carried out (Ericsson & Simon,
1984, 1993; Olson & Biolsi, 199 1). Second, the goal of this article is to provide a practical guide;
combining a guide along with an analysis and synthesis of the literature would be beyond the scope of
this article.

Quantifying Qualitative Analyses of Verbal Data: A Practical Guide 2
An alternative way to write such a guide, taken in this article, is to describe and deconstruct the
methods that I have developed and used in my own research over the past decade or so. Reflecting on
these case studies can reveal the techniques and assumptions that underlie the research method, which
will henceforth be called verbal analysis. Thus, no attempt is made to relate aspects of the verbal
analysis method (in terms of degree of overlap or whether the verbal analysis method complements
some other methods) to those others have used, with the exception of one very obvious case. The
exception is the long-standing work on protocol analysis, initiated by the information-processing
approach (Ericsson & Simon, 1984,1993; Newell & Simon, 1972). Contrasting the protocol analysis
method with the verbal analysis method is necessary because the two methods, up to a point, share a
number of surface similarities in the mechanics of the coding. There are numerous other equally
well-thought-out methods in the literature using verbal analysis, such as the proposition-discourse
analysis of Patel and Groen (1986), conversation analysis of Fox (1991), inferential flow analysis of
Salter (1983) and others. Thus, the method described here is but one approach to analyzing verbal
data and no claim is being made about its value in comparison to other related methods.
Why is a practical guide on my personal method of any use? First, as lucidly pointed out by
Schofield and Anderson (1987), both qualitative and quantitative analyses have shortcomings and
strengths, thus some kind of method that can integrate elements of both methods seems desirable,
especially for answering complex questions such as learning in context. Thus, laying out and sharing
one such method seems useful, even if it is not the best one. Second, because analyses of verbal data
are often complex, the techniques for doing them are usually opaque. Thus, an explicit guide seems
useful at least as a starting point to overcome that opaqueness, even if this guide illustrates a special
case.
This article has basically three sections. The first section raises three introductory issues: my
theoretical bias, comparison of this method with the protocol analysis method, and different
approaches to integrating quantitative and qualitative analyses. The second section describes the
mechanics of the method, which has been decomposed into eight steps (henceforth, the specific steps
will be referred to as the technique, and the general method is referred to as the verbal analysis
method). The third section provides additional recommendations and technical details and addresses
some remaining questions and caveats. The article closes with a concluding summary.
INTRODUCTION TO VERBAL ANALYSIS
Verbal analysis is a methodology for quantifying the subjective or qualitative coding of the contents
of verbal utterances. In verbal analysis, one tabulates, counts, and draws relations between the
occurrences of different kinds of utterances to reduce the subjectiveness of qualitative coding. Verbal
analysis has been used, for example, to code explanations of what one understands as one reads a text
sentence-by-sentence, to see whether an explanation is an inference, a monitoring statement, or
some other irrelevant comment (Chi, de Leeuw, Chiu, & LaVancher, 1994). Such quantification of
qualitative coding is not the same as direct counting methods whereby a researcher picks out aspects
of the qualitative data that can be quantified directly, such as counting the occurrence of a given word
in a newspaper article (Weber, 1985). Verbal analysis is also to be differentiated from methods
whereby a researcher undertakes qualitative observations in a messy context, but then analyzes only
the quantitative data from that messy situation. For example, suppose one observes how the
introduction of new technological equipment effects the operation in a trauma unit, but then
ultimately analyzes the mortality rate or the cost of treatment per patient as a consequence of
having that new technology. In both of these cases, either no subjective qualitative coding was
entailed or no qualitative data were actually used. Although such methods sometimes do allow analysis
of the content (e.g., one has to read the content of the newspaper article to decide which word should
be coded), verbal analysis, in addition, allows for coding that does not require a direct correspondence
between the content word(s) uttered and the coding category.
The verbal analysis method is embedded in research that tries to understand cognition, and in
particular, the kind of knowledge one gains from learning. However, the mechanics of the method
may be adapted to the study of various noncognitive issues (such as social, motivational, and
behavioral) and may also be used with observational and video data, rather than strictly with verbal
data. However, every approach, as used by a specific investigator, has a theoretical bias that is built
into the method. Let me describe that bias first.