Sign up & Download
Sign in

Toward more substantively meaningful automated essay scoring

by A Ben-Simon
The Journal of Technology, Learning ()
  • ISSN: 1540-2525

Abstract

Tis study evaluated a “substantively driven” method for scoring NAEP writing assess- ments automatically. Te study used variations of an existing commercial program, e-rater®, to compare the performance of three approaches to automated essay scoring: a brute-empirical approach in which variables are selected and weighted solely accord- ing to statistical criteria, a hybrid approach in which a fixed set of variables more closely tied to the characteristics of good writing was used but the weights were still statisti- cally determined, and a substantively driven approach in which a fixed set of variables was weighted according to the judgments of two independent committees of writing experts. Te research questions concerned (a) the reproducibility of weights across writing experts, (b) the comparison of scores generated by the three automated approaches, and (c) the extent to which models developed for scoring one NAEP prompt generalize to other NAEP prompts of the same genre. Data came from the 2002 NAEP Writing Online study and from the main NAEP 2002 writing assessment. Results showed that, in carrying out the substantively driven approach, experts initially assigned weights to writing dimensions that were highly similar across committees but that diverged from one another after committee 1 was shown the empirical weights for possible use in its judgments and committee 2 was not shown those weights. Te substan- tively driven approach based on the judgments of committee 1 generally did not operate in a markedly different way from the brute empirical or hybrid approaches in most of the analyses conducted. In contrast, many consistent differences with those approaches were observed for the substantively driven approach based on the judgments of com- mittee 2. Tis study suggests that empirical weights might provide a useful starting point for expert committees, with the understanding that the weights be moderated only somewhat to bring them more into line with substantive considerations. Under such circumstances, the results may turn out to be reasonable, though not necessarily as highly related to human ratings as statistically optimal approaches would produce.

Cite this document (BETA)

Readership Statistics

21 Readers on Mendeley
by Discipline
 
 
 
by Academic Status
 
24% Ph.D. Student
 
24% Researcher (at a non-Academic Institution)
 
10% Doctoral Student
by Country
 
10% Australia
 
10% United States
 
5% Japan

Sign up today - FREE

Mendeley saves you time finding and organizing research. Learn more

  • All your research in one place
  • Add and import papers easily
  • Access it anywhere, anytime

Start using Mendeley in seconds!

Already have an account? Sign in