Swan and ArabicMTEB: Dialect-Aware, Arabic-Centric, Cross-Lingual, and Cross-Cultural Embedding Models and Benchmarks

TL;DR

Swan introduces dialect-aware, Arabic-centric embeddings with two variants (Swan-Small and Swan-Large) and the ArabicMTEB benchmark to assess cross-lingual, dialectal, domain-specific, and cultural performance across 94 datasets and eight tasks. The models are trained on a large, diverse mix of MSA, dialectal, and cross-lingual data using a two-stage strategy with instruction-enhanced queries and hard negatives, optimized by an InfoNCE loss. ArabicMTEB provides a broad, multi-dimensional evaluation framework, including dialectal and cultural forks (ArabicMTEB-Lite and Cultural ArabicMTEB). Results show Swan-Large achieves state-of-the-art performance on most Arabic tasks while Swan-Small consistently outperforms Multilingual-E5 baselines, with substantial cost advantages for large-scale deployment. The work delivers open resources (models and benchmark) to advance Arabic NLP research and practical applications across diverse dialects and cultural contexts.

Abstract

We introduce {\bf Swan}, a family of embedding models centred around the Arabic language, addressing both small-scale and large-scale use cases. Swan includes two variants: Swan-Small, based on ARBERTv2, and Swan-Large, built on ArMistral, a pretrained Arabic large language model. To evaluate these models, we propose ArabicMTEB, a comprehensive benchmark suite that assesses cross-lingual, multi-dialectal, multi-domain, and multi-cultural Arabic text embedding performance, covering eight diverse tasks and spanning 94 datasets. Swan-Large achieves state-of-the-art results, outperforming Multilingual-E5-large in most Arabic tasks, while the Swan-Small consistently surpasses Multilingual-E5-base. Our extensive evaluations demonstrate that Swan models are both dialectally and culturally aware, excelling across various Arabic domains while offering significant monetary efficiency. This work significantly advances the field of Arabic language modelling and provides valuable resources for future research and applications in Arabic natural language processing. Our models and benchmark are available at our GitHub page: \href{https://github.com/UBC-NLP/swan}{https://github.com/UBC-NLP/swan}

Figures (4)

• Figure 1: Overview of our ArabicMTEB benchmark tasks, covering clustering, retrieval, reranking, classification, semantic similarity, pair classification, cross-lingual retrieval, and bitext mining.
• Figure 2: Methodology to generate our synthetic data.
• Figure 3: Generation pipeline for our domain specific ArabicMTEB.
• Figure 4: Latency vs Performance.