ViDeBERTa: A powerful pre-trained language model for Vietnamese

5Citations
Citations of this article
18Readers
Mendeley users who have this article in their library.

Abstract

This paper presents ViDeBERTa, a new pre-trained monolingual language model for Vietnamese, with three versions - ViDeBERTaxsmall, ViDeBERTabase, and ViDeBERTalarge, which are pre-trained on a large-scale corpus of high-quality and diverse Vietnamese texts using DeBERTa architecture. Although many successful pre-trained language models based on Transformer have been widely proposed for the English language, there are still few pre-trained models for Vietnamese, a low-resource language, that perform good results on downstream tasks, especially Question answering. We fine-tune and evaluate our model on three important natural language downstream tasks, Part-of-speech tagging, Named-entity recognition, and Question answering. The empirical results demonstrate that ViDeBERTa with far fewer parameters surpasses the previous state-of-the-art models on multiple Vietnamese-specific natural language understanding tasks. Notably, ViDeBERTabase with 86M parameters, which is only about 23% of PhoBERTlarge with 370M parameters, still performs the same or better results than the previous state-of-the-art model. Our ViDeBERTa models are available at: https://github.com/HySonLab/ViDeBERTa.

Cite

CITATION STYLE

APA

Tran, C. D., Pham, N. H., Nguyen, A., Hy, T. S., & Vu, T. (2023). ViDeBERTa: A powerful pre-trained language model for Vietnamese. In EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Findings of EACL 2023 (pp. 1041–1048). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2023.findings-eacl.79

Register to see more suggestions

Mendeley helps you to discover research relevant for your work.

Already have an account?

Save time finding and organizing research with Mendeley

Sign up for free