Non-neural Models Matter: A Re-evaluation of Neural Referring Expression Generation Systems

9Citations
Citations of this article
34Readers
Mendeley users who have this article in their library.

Abstract

In recent years, neural models have often outperformed rule-based and classic Machine Learning approaches in NLG. These classic approaches are now often disregarded, for example when new neural models are evaluated. We argue that they should not be overlooked, since for some tasks, well-designed non-neural approaches achieve better performance than neural ones. In this paper, the task of generating referring expressions in linguistic context is used as an example. We examined two very different English datasets (WEBNLG and WSJ), and evaluated each algorithm using both automatic and human evaluations. Overall, the results of these evaluations suggest that rule-based systems with simple rule sets achieve on-par or better performance on both datasets compared to state-of-the-art neural REG systems. In the case of the more realistic dataset, WSJ, a machine learning-based system with well-designed linguistic features performed best. We hope that our work can encourage researchers to consider non-neural models in future.

References Powered by Scopus

Long Short-Term Memory

76929Citations
N/AReaders
Get full text

Generation and comprehension of unambiguous object descriptions

898Citations
N/AReaders
Get full text

Survey of the state of the art in natural language generation: Core tasks, applications and evaluation

550Citations
N/AReaders
Get full text

Cited by Powered by Scopus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text

60Citations
N/AReaders
Get full text

A survey on semantic processing techniques

30Citations
N/AReaders
Get full text

A Multi-task Learning Model for Gold-two-mention Co-reference Resolution

5Citations
N/AReaders
Get full text

Register to see more suggestions

Mendeley helps you to discover research relevant for your work.

Already have an account?

Cite

CITATION STYLE

APA

Same, F., Chen, G., & van Deemter, K. (2022). Non-neural Models Matter: A Re-evaluation of Neural Referring Expression Generation Systems. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (Vol. 1, pp. 5554–5567). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2022.acl-long.380

Readers' Seniority

Tooltip

PhD / Post grad / Masters / Doc 4

40%

Researcher 3

30%

Professor / Associate Prof. 2

20%

Lecturer / Post doc 1

10%

Readers' Discipline

Tooltip

Computer Science 8

62%

Linguistics 3

23%

Neuroscience 1

8%

Engineering 1

8%

Save time finding and organizing research with Mendeley

Sign up for free