Automatic assessment of open ended questions with a BLEU-inspired algorithm and shallow NLP

Abstract

This paper compares the accuracy of several variations of the BLEU algorithm when applied to automatically evaluating student essays. The different configurations include closed-class word removal, stemming, two baseline word-sense disambiguation procedures, and translating the texts into a simple semantic representation. We also prove empirically that the accuracy is kept when the student answers are translated automatically. Although none of the representations clearly outperform the others, some conclusions are drawn from the results.