Recent years have seen increasing interest in automatic metrics for the evaluation of generation systems. When a system can generate syntactic variation, automatic evaluation becomes more difficult. In this paper, we compare the performance of several automatic evaluation metrics using a corpus of automatically generated paraphrases. We show that these evaluation metrics can at least partially measure adequacy (similarity in meaning), but are not good measures of fluency (syntactic correctness). We make several proposals for improving the evaluation of generation systems that produce variation. © Springer-Verlag Berlin Heidelberg 2005.
CITATION STYLE
Stent, A., Marge, M., & Singhai, M. (2005). Evaluating evaluation methods for generation in the presence of variation. In Lecture Notes in Computer Science (Vol. 3406, pp. 341–351). Springer Verlag. https://doi.org/10.1007/978-3-540-30586-6_38
Mendeley helps you to discover research relevant for your work.