Manual and Automatic Identification of Similar Arguments in EFL Learner Essays

Abstract

Argument mining typically focuses on identifying argumentative units such as claim, position, evidence etc. in texts. In an educational setting, e.g. when teachers grade students’ essays, they may in addition benefit from information about the content of the arguments being used. We thus present a pilot study on the identification of similar arguments in a set of essays written by English-as-a-foreign-language (EFL) students. In a manual annotation study, we show that human annotators are able to assign sentences to a set of 26 reference arguments with a rather high agreement of κ > .70. In a set of experiments based on (a) unsupervised clustering and (b) supervised machine learning, we find that both approaches perform rather poorly on this task, but can be moderately improved by using a set of six meta classes instead of the more fine-grained argument distinction.