ArabicNLU 2024: The First Arabic Natural Language Understanding Shared Task
Mohammed Khalilia, Sanad Malaysha, Reem Suwaileh, Mustafa Jarrar, Alaa Aljabari, Tamer Elsayed, Imed Zitouni
TL;DR
ArabicNLU 2024 tackles two core Arabic NLU challenges, WSD and LMD, by releasing SALMA and IDRISI-DA datasets and organizing a dedicated shared task. Discriminative, fine-tuned models outperform zero-shot generative approaches for WSD, while LMD results reveal strong baselines from geocoding pipelines but also Arabic-specific bottlenecks due to dialectal variation and translation steps. The work provides public resources, baselines, and insights into model limitations and data needs for Arabic NLU, emphasizing the need for dialect-aware datasets and Arabic-tailored LLMs to enable robust, real-world Arabic NLP applications. Overall, the shared task establishes a benchmark framework and practical resources to accelerate progress in Arabic word and location disambiguation.
Abstract
This paper presents an overview of the Arabic Natural Language Understanding (ArabicNLU 2024) shared task, focusing on two subtasks: Word Sense Disambiguation (WSD) and Location Mention Disambiguation (LMD). The task aimed to evaluate the ability of automated systems to resolve word ambiguity and identify locations mentioned in Arabic text. We provided participants with novel datasets, including a sense-annotated corpus for WSD, called SALMA with approximately 34k annotated tokens, and the IDRISI-DA dataset with 3,893 annotations and 763 unique location mentions. These are challenging tasks. Out of the 38 registered teams, only three teams participated in the final evaluation phase, with the highest accuracy being 77.8% for WSD and the highest MRR@1 being 95.0% for LMD. The shared task not only facilitated the evaluation and comparison of different techniques, but also provided valuable insights and resources for the continued advancement of Arabic NLU technologies.