Implicit N-grams Induced by Recurrence

2Citations
Citations of this article
37Readers
Mendeley users who have this article in their library.

Abstract

Although self-attention based models such as Transformers have achieved remarkable successes on natural language processing (NLP) tasks, recent studies reveal that they have limitations on modeling sequential transformations (Hahn, 2020), which may prompt re-examinations of recurrent neural networks (RNNs) that demonstrated impressive results on handling sequential data. Despite many prior attempts to interpret RNNs, their internal mechanisms have not been fully understood, and the question on how exactly they capture sequential features remains largely unclear. In this work, we present a study that shows there actually exist some explainable components that reside within the hidden states, which are reminiscent of the classical n-grams features. We evaluated such extracted explainable features from trained RNNs on downstream sentiment analysis tasks and found they could be used to model interesting linguistic phenomena such as negation and intensification. Furthermore, we examined the efficacy of using such n-gram components alone as encoders on tasks such as sentiment analysis and language modeling, revealing they could be playing important roles in contributing to the overall performance of RNNs. We hope our findings could add interpretability to RNN architectures, and also provide inspirations for proposing new architectures for sequential data.

References Powered by Scopus

Long Short-Term Memory

76931Citations
N/AReaders
Get full text

GloVe: Global vectors for word representation

26880Citations
N/AReaders
Get full text

Learning phrase representations using RNN encoder-decoder for statistical machine translation

11647Citations
N/AReaders
Get full text

Cited by Powered by Scopus

TAPIR: Learning Adaptive Revision for Incremental Natural Language Understanding with a Two-Pass Model

4Citations
N/AReaders
Get full text

Large Language Model Instruction Following: A Survey of Progresses and Challenges

0Citations
N/AReaders
Get full text

Register to see more suggestions

Mendeley helps you to discover research relevant for your work.

Already have an account?

Cite

CITATION STYLE

APA

Sun, X., & Lu, W. (2022). Implicit N-grams Induced by Recurrence. In NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference (pp. 1624–1639). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2022.naacl-main.117

Readers' Seniority

Tooltip

PhD / Post grad / Masters / Doc 7

54%

Researcher 5

38%

Lecturer / Post doc 1

8%

Readers' Discipline

Tooltip

Computer Science 14

78%

Linguistics 2

11%

Neuroscience 1

6%

Agricultural and Biological Sciences 1

6%

Save time finding and organizing research with Mendeley

Sign up for free