Implementing a Contributory Scoring Approach for the GRE ® Analytical Writing Section: A Comprehensive Empirical Investigation

Abstract

In this research report, we present an empirical argument for the use of a contributory scoring approach for the 2‐essay writing assessment of the analytical writing section of the GRE ® test in which human and machine scores are combined for score creation at the task and section levels. The approach was designed to replace a currently operational all‐human check scoring approach in which machine scores are used solely as quality‐control checks to determine when additional human ratings are needed due to unacceptably large score discrepancies. We use data from 6 samples of essays collected from test takers during operational administrations and special validity studies to empirically evaluate 6 different score computation methods. During the presentation of our work, we critically discuss key methodological design decisions and underlying rationales for these decisions. We close the report by discussing how the research methodology is generalizable to other testing programs and use contexts. Report Number: ETS RR‐17–14