Skip to content

Toward More Substantively Meaningful Automated Essay Scoring

by Anat Ben-Simon, Randy Elliot Bennett
The Journal of Technology, Learning, and Assessment ()
  • ISSN: 1540-2525


Tis study evaluated a “substantively driven” method for scoring NAEP writing assess- ments automatically. Te study used variations of an existing commercial program, e-rater®, to compare the performance of three approaches to automated essay scoring: a brute-empirical approach in which variables are selected and weighted solely accord- ing to statistical criteria, a hybrid approach in which a fixed set of variables more closely tied to the characteristics of good writing was used but the weights were still statisti- cally determined, and a substantively driven approach in which a fixed set of variables was weighted according to the judgments of two independent committees of writing experts. Te research questions concerned (a) the reproducibility of weights across writing experts, (b) the comparison of scores generated by the three automated approaches, and (c) the extent to which models developed for scoring one NAEP prompt generalize to other NAEP prompts of the same genre. Data came from the 2002 NAEP Writing Online study and from the main NAEP 2002 writing assessment. Results showed that, in carrying out the substantively driven approach, experts initially assigned weights to writing dimensions that were highly similar across committees but that diverged from one another after committee 1 was shown the empirical weights for possible use in its judgments and committee 2 was not shown those weights. Te substan- tively driven approach based on the judgments of committee 1 generally did not operate in a markedly different way from the brute empirical or hybrid approaches in most of the analyses conducted. In contrast, many consistent differences with those approaches were observed for the substantively driven approach based on the judgments of com- mittee 2. Tis study suggests that empirical weights might provide a useful starting point for expert committees, with the understanding that the weights be moderated only somewhat to bring them more into line with substantive considerations. Under such circumstances, the results may turn out to be reasonable, though not necessarily as highly related to human ratings as statistically optimal approaches would produce.

Cite this document (BETA)

Readership Statistics

32 Readers on Mendeley
by Discipline
44% Social Sciences
34% Computer Science
16% Linguistics
by Academic Status
25% Researcher
25% Student > Ph. D. Student
13% Student > Master
by Country
6% Germany
6% Australia
6% Japan

Sign up today - FREE

Mendeley saves you time finding and organizing research. Learn more

  • All your research in one place
  • Add and import papers easily
  • Access it anywhere, anytime

Start using Mendeley in seconds!

Sign up & Download

Already have an account? Sign in