This paper presents ViDeBERTa, a new pre-trained monolingual language model for Vietnamese, with three versions - ViDeBERTaxsmall, ViDeBERTabase, and ViDeBERTalarge, which are pre-trained on a large-scale corpus of high-quality and diverse Vietnamese texts using DeBERTa architecture. Although many successful pre-trained language models based on Transformer have been widely proposed for the English language, there are still few pre-trained models for Vietnamese, a low-resource language, that perform good results on downstream tasks, especially Question answering. We fine-tune and evaluate our model on three important natural language downstream tasks, Part-of-speech tagging, Named-entity recognition, and Question answering. The empirical results demonstrate that ViDeBERTa with far fewer parameters surpasses the previous state-of-the-art models on multiple Vietnamese-specific natural language understanding tasks. Notably, ViDeBERTabase with 86M parameters, which is only about 23% of PhoBERTlarge with 370M parameters, still performs the same or better results than the previous state-of-the-art model. Our ViDeBERTa models are available at: https://github.com/HySonLab/ViDeBERTa.
CITATION STYLE
Tran, C. D., Pham, N. H., Nguyen, A., Hy, T. S., & Vu, T. (2023). ViDeBERTa: A powerful pre-trained language model for Vietnamese. In EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Findings of EACL 2023 (pp. 1041–1048). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2023.findings-eacl.79
Mendeley helps you to discover research relevant for your work.