Jawaher: A Multidialectal Dataset of Arabic Proverbs for LLM Benchmarking

TL;DR

Jawaher introduces a multidialectal Arabic proverb benchmark (10,037 proverbs across 20 dialects) with idiomatic English translations and explanations in English and Arabic to evaluate LLMs' handling of figurative language. The study conducts zero-shot evaluations on open- and closed-source multilingual LLMs, using automatic metrics (BLEURT, BERTScore) and human judgments to assess translation and explanation tasks. Findings show models excel at idiomatic translations but struggle to deliver culturally nuanced, context-rich explanations, particularly for Arabic proverbs; closed models generally outperform open ones, though gaps remain in cultural depth and sensitivity. The benchmark highlights a cultural gap in LLMs and motivates continued data expansion and model refinement to improve figurative-language understanding across diverse dialects.

Abstract

Recent advancements in instruction fine-tuning, alignment methods such as reinforcement learning from human feedback (RLHF), and optimization techniques like direct preference optimization (DPO) have significantly enhanced the adaptability of large language models (LLMs) to user preferences. However, despite these innovations, many LLMs continue to exhibit biases toward Western, Anglo-centric, or American cultures, with performance on English data consistently surpassing that of other languages. This reveals a persistent cultural gap in LLMs, which complicates their ability to accurately process culturally rich and diverse figurative language such as proverbs. To address this, we introduce Jawaher, a benchmark designed to assess LLMs' capacity to comprehend and interpret Arabic proverbs. Jawaher includes proverbs from various Arabic dialects, along with idiomatic translations and explanations. Through extensive evaluations of both open- and closed-source models, we find that while LLMs can generate idiomatically accurate translations, they struggle with producing culturally nuanced and contextually relevant explanations. These findings highlight the need for ongoing model refinement and dataset expansion to bridge the cultural gap in figurative language processing.

Figures (9)

• Figure 1: Overview of the Jawaher benchmark, featuring Arabic proverbs along with their idiomatic translations and explanations in both English and Arabic; for instance, the Libyan proverb ولد الحنش ثعبان whose literal meaning is "the son of a snake is a serpent", is explained as conveying the idea that "like father, like son in terms of bad behavior and twisted character".
• Figure 2: Choropleth map showing the geographical distribution of Arabic varieties covered in Jawaher. Color intensity represents the percentage of proverbs collected from each region, with darker shades indicating higher concentrations.
• Figure 3: UMAP visualization of proverb embeddings, showing clusters corresponding to four key themes: Food, Mother, Children, and Eyes. Proverbs are grouped based on thematic and cultural similarities across various Arab dialects.
• Figure 4: KDE plots of proverb embeddings from various Arabic dialects, highlighting both shared cultural themes and regional variations among Gulf, Levantine, North African, Arabian Peninsula and Nile Basin dialects.
• Figure 5: Designed prompt used to test model's performance on Jawaher.