Dialect-Aware, Arabic-Centric, Cross-Lingual, and Cross-Cultural Embedding Models and Benchmarks

Abstract

We introduce Swan, a family of embedding models centered on Arabic, designed for both small-scale and large-scale applications. Swan comprises two variants: Swan-Small, built on ARBERTv2, and Swan-Large, based on ArMistral, a pretrained Arabic large language model. To evaluate our models, we propose a comprehensive benchmark suite, dubbed ArabicMTEB, that assesses cross-lingual, multi-dialectal, multi-domain, and multi-cultural Arabic text embedding performance. ArabicMTEBcovers eight diverse tasks sourced from 94 datasets. Swan-Large achieves state-of-the-art results, outperforming Multilingual-E5-large in most Arabic tasks, while Swan-Small consistently surpasses Multilingual-E5-base. Our extensive evaluations show that Swan models are both dialectally and culturally aware, achieving strong performance across diverse Arabic domains while maintaining significant cost efficiency. This work significantly advances the field of Arabic language modelling and provides valuable resources for future research and applications in Arabic NLP. Our models and benchmark are available at our GitHub page: https://github.com/UBC-NLP/swan.