On-the-Fly Attention Modulation for Neural Generation

6Citations
Citations of this article
66Readers
Mendeley users who have this article in their library.

Abstract

Despite considerable advancements with deep neural language models (LMs), neural text generation still suffers from degeneration: the generated text is repetitive, generic, self-contradictory, and often lacks commonsense. Our analyses on sentence-level attention patterns in LMs reveal that neural degeneration may be associated with insufficient learning of task-specific characteristics by the attention mechanism. This finding motivates on-the-fly attention modulation- a simple but effective method that enables the injection of priors into attention computation during inference. Automatic and human evaluation results on three text generation benchmarks demonstrate that attention modulation helps LMs generate text with enhanced fluency, creativity, and commonsense reasoning, in addition to significantly reduce sentence-level repetition.

Cite

CITATION STYLE

APA

Dong, Y., Bhagavatula, C., Lu, X., Hwang, J. D., Bosselut, A., Cheung, J. C. K., & Choi, Y. (2021). On-the-Fly Attention Modulation for Neural Generation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 (pp. 1261–1274). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2021.findings-acl.107

Register to see more suggestions

Mendeley helps you to discover research relevant for your work.

Already have an account?

Save time finding and organizing research with Mendeley

Sign up for free