VACASPATI: A Diverse Corpus of Bangla Literature

Abstract

Bangla (or Bengali) is the fifth most spoken language globally; yet, the state-of-the-art NLP in Bangla is lagging for even simple tasks such as lemmatization, POS tagging, etc. This is partly due to lack of a varied quality corpus. To alleviate this need, we build VĀCASPATI, a diverse corpus of Bangla literature. The literary works are collected from various websites; only those works that are publicly available without copyright violations or restrictions are collected. We believe that published literature captures the features of a language much better than newspapers, blogs or social media posts which tend to follow only a certain literary pattern and, therefore, miss out on language variety and vocabulary. Our corpus VĀCASPATI is varied from multiple aspects, including type of composition, topic, author, time, space, etc. It contains more than 11 million sentences and 115 million words. We have also built a word embedding model, VĀC-FT, using FastText from VĀCASPATI as well as trained an Electra model, VĀC-BERT, using the corpus. VĀC-BERT has far fewer parameters and requires only a fraction of resources compared to other state-of-the-art transformer models and yet performs either better or similar on various downstream tasks. Similarly, VĀC-FT outperforms other FastText-based models on multiple downstream tasks. We also demonstrate the efficacy of VĀCASPATI as a corpus by showing that similar models built from other corpora are not as effective.