Submitted 06.02.2002 © 2002 UMass Physics Education Research Group
Measuring and Modeling Physics Students’
Conceptual Knowledge Structures Through
Term Association Times
Ian D. Beatty
University of Massachusetts, Amherst, MA, USA1
William J. Gerace
University of Massachusetts, Amherst, MA, USA
Robert J. Dufresne
University of Massachusetts, Amherst, MA, USA
Traditional problem-based exams are not efficient instruments for
assessing the “structure” of physics students’ conceptual knowledge or
for providing diagnostically detailed feedback to students and
teachers. We present the Free Term Entry task, a candidate assessment
instrument for exploring the connections between concepts in a
student’s understanding of a subject. In this task, a student is given a
general topic area and asked to respond with as many terms from the
topic area as possible in a given time; the “thinking time” between
each term-entry event is recorded along with the response terms. The
task was given to students from two different introductory physics
classes. Response term thinking times were found to correlate with the
strength of the association between two concepts. In addition, sets of
thinking times from the task show distinct, characteristic patterns
which might prove valuable for student assessment. We propose a
quantitative dynamical model named the Matrix Walk Model which is
able to match many aspects of the observed data. One particular
feature of the data — a distinct “spike” superimposed on the otherwise
log-normal distribution of most thinking time sets — has not been fit.
The spike, other patterns observed in the data, and the proposed
phenomenological model could all benefit from a grounding in
cognitive theory.
Keywords: physics, assessment, conceptual knowledge, concept map,
modeling, term association.

1 Address for correspondence: Dr. Ian Beatty, Physics Department, Lederle Graduate
Research Tower 417A, University of Massachusetts, Amherst MA 01003-4525 USA. E-mail:
beatty@physics.umass.edu. Home page: http://umperg.physics.umass.edu/

2 Measuring and Modeling Physics Students’ Conceptual Knowledge…
I. Introduction
“Assessment drives pedagogy” is an oft-heard phrase in educational
research circles. It derives from the tendency of grade-conscious students to
optimize their learning for exam results, and of results-conscious teachers to
optimize their curriculum for class performance on assessments. It holds
true for the style as well as the content of assessments: if tests target rote
knowledge, students will perceive that as the objective of instruction; if they
target conceptual reasoning and transfer, students will more likely focus on
those.
Therefore, assessment is presently a lively topic of educational research
(Nichols, Chipman, & Brennan, 1995). Much of this research is devoted to
developing cognitively diagnostic, formative assessments. “Cognitively
diagnostic” means the assessments can be used to characterize the state of
knowledge of individual students with enough detail to guide students’
subsequent learning efforts and teachers’ subsequent interventions, as
opposed to merely characterizing the learning of a population of students
(e.g., for curriculum evaluation) benchmarking students’ gross level of
mastery (e.g., for placement decisions). “Formative” means the assessments
are used in an ongoing way during instruction to guide and enhance
teaching and learning, rather than after instruction to evaluate success.
Cognitively diagnostic assessment requires two foundations: instruments
that can probe and gather data on relevant features of a student’s state of
knowledge, and a model of topic knowledge and learning by which the data
may be interpreted. Development of these two foundations is necessarily
interdependent, because any proposed probe instrument can only be
justified through the interpretability of the data it yields, and interpretation
requires a model; and a model is justified by its ability to explain observed
data. This chicken-and-egg relationship is common to all young research
fields.
Acknowledging this, we have attempted to simultaneously develop
assessment methods for probing introductory physics students’ state of
knowledge at a fundamental level, and a complementary quantitative model
of conceptual knowledge. In a previous paper (Beatty & Gerace, 2001) we
proposed a family of computer-based tasks as assessment instruments, and
presented data suggesting that the tasks are sensitive to relevant aspects of
students’ conceptual knowledge structure (CKS). In this paper we focus on one
of those tasks, analyzing in more detail the data it can provide. In addition,
we suggest a simple quantitative dynamical model of CKS and how it is
accessed which can “explain” the observed data.
It is not our intent here to suggest new research tools for basic cognitive
science research, nor to propose a mature, ready-to-use assessment
instrument, nor to present a complete or fundamental cognitive model.
Rather, it is to explore the potential of a new (in this context) assessment
approach for eliciting information about students’ CKS, and to demonstrate
how the data produced can be quantitatively modeled. That is, we are
attempting a “proof-of-concept” venture. Our results are for the domain of
introductory physics, but we see no reason why they might not be