MaroBERTa: Multilabel Classification Language Model for Darija Newspaper

Abstract

A large amount of valuable digital text, audio and video data is available on the web. Thus, a large application of machine learning based on Natural Language Processing (NLP) has taken advantage of these opportunities. Transformers, especially Bidirectional Encoder Representation from Transformers (BERT) based models, have become the state-of-the-art for downstream NLP tasks. Non-normalized languages such as Moroccan Arabic, also known as Darija, increases the complexity of natural language processing. Furthermore, Text written in Darija does not have a standard spelling, and there is a lack of resources, especially for multilabel classification. In this paper, we introduced a multilabel classification model for Moroccan Arabic (Darija) newspapers. Firstly, we created a dataset from 400.000 collected newspaper articles with their titles, written in darija and pre-trained our model: MaroBERTa. Secondly, we implemented a crowd-sourcing platform to help create a novel corpus called Darija Multilabel Dataset for News classification (DMDNews). This dataset contains 28 different classes representing the most frequent topics in Moroccan newspapers. Finally, we fine-tune MaroBERTa and two multilingual models (AraBERT and CAMelBert) for the multilabel classification task using the DMDNews. Experiments shows that our dedicated pretrained Darija model -MaroBERTa- outperforms the existing multilingual models despite of the large amount of data they have been trained on.