Abstract
This paper presents an investigation aimed at studying how the linguistic structure of a sentence affects the perplexity of two of the most popular Neural Language Models (NLMs), BERT and GPT-2. We first compare the sentence–level likelihood computed with BERT and the GPT-2’s perplexity showing that the two metrics are correlated. In addition, we exploit linguistic features capturing a wide set of morpho-syntactic and syntactic phenomena showing how they contribute to predict the perplexity of the two NLMs.
Register to see more suggestions
Mendeley helps you to discover research relevant for your work.
Cite
CITATION STYLE
Miaschi, A., Brunato, D., Dell’Orletta, F., & Venturi, G. (2021). What Makes My Model Perplexed? A Linguistic Investigation on Neural Language Models Perplexity. In Deep Learning Inside Out: 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, DeeLIO 2021 - Proceedings, co-located with the Annual Conference of the North American Chapter of the Association for Computational Linguistics, NAACL-HLT 2021 (pp. 40–47). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2021.deelio-1.5
Readers' Seniority
Readers' Discipline
