Building e‐rater ® Scoring Models Using Machine Learning Methods

Abstract

The e‐rater ® automated scoring engine used at Educational Testing Service ( ETS ) scores the writing quality of essays. In the current practice, e‐rater scores are generated via a multiple linear regression ( MLR ) model as a linear combination of various features evaluated for each essay and human scores as the outcome variable. This study evaluates alternative scoring models based on several additional machine learning algorithms, including support vector machines ( SVM ), random forests ( RF ), and k‐ nearest neighbor regression ( k‐ NN ). The results suggest that models based on the SVM algorithm outperform MLR models in predicting human scores. Specifically, SVM ‐based models yielded the highest agreement between human and e‐rater scores. Furthermore, compared with MLR , SVM ‐based models improved the agreement between human and e‐rater scores at the ends of the score scale. In addition, the high correlation between SVM ‐based e‐rater scores with external measures such as examinee's scores on the other parts of the test provided some validity evidence for SVM ‐based e‐rater scores. Future research is encouraged to explore the generalizability of these findings. Report Number: ETS RR‐16–04