A Self-Supervised Integration Method of Pretrained Language Models and Word Definitions

2Citations
Citations of this article
12Readers
Mendeley users who have this article in their library.

Abstract

We investigate the representation of pretrained language models and humans, using the idea of word definition modeling-how well a word is represented by its definition, and vice versa. Our analysis shows that a word representation in pretrained language models does not successfully map its human-written definition and its usage in example sentences. We then present a simple method DefBERT that integrates pretrained models with word semantics in dictionaries. We show its benefits on newly-proposed tasks of definition ranking and definition sense disambiguation. Furthermore, we present the results on standard word similarity tasks and short text classification tasks where models are required to encode semantics with only a few words. The results demonstrate the effectiveness of integrating word definitions and pretrained language models.

Cite

CITATION STYLE

APA

Jo, H. (2023). A Self-Supervised Integration Method of Pretrained Language Models and Word Definitions. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (pp. 14–26). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2023.findings-acl.2

Register to see more suggestions

Mendeley helps you to discover research relevant for your work.

Already have an account?

Save time finding and organizing research with Mendeley

Sign up for free