A Reference-less Evaluation Metric Based on Grammaticality, Fluency, and Meaning Preservation in Grammatical Error Correction

Abstract

In grammatical error correction (GEC), the automatic evaluation of system performance is thought to be an essential driving force. Previous methods for automated system assessment require gold-standard references, which have to be created manually and thus tend to be both expensive and limited in coverage. To address this problem, a reference-less approach has recently emerged; however, previous reference-less metrics, which only consider the grammaticality of system outputs, have not performed as well as reference-based metrics. In this study, we explore the potential of extending a prior grammaticality-based method to establish a reference-less evaluation method for GEC systems. We empirically show that a reference-less metric that combines both fluency and meaning preservation with grammaticality provides a better estimate of manual scores than that of commonly used reference-based met-rics. Additionally, we show that the reference-less metric can provide appropriate evaluation at the sentence-level and that it can be applied to GEC systems. † ,