MasalBench: A Benchmark for Contextual and Cross-Cultural Understanding of Persian Proverbs in LLMs

TL;DR

MasalBench targets Persian proverb understanding in multilingual LLMs by introducing two tasks: contextual understanding with 1,000 dialogue-based MCQs and cross-cultural understanding with 700 binary questions linking Persian proverbs to English equivalents. The dataset is constructed from Foote Koozegari and generated with Gemini 2.5 Pro, with manual verification to ensure cultural authenticity. Eight multilingual LLMs are evaluated, showing high contextual comprehension but weaker cross-cultural transfer, and an error pattern dominated by plausible distractors rather than literal mistakes. The benchmark highlights the need for cultural grounding and analogical reasoning in cross-language proverb understanding and provides a publicly available resource to advance cross-lingual and culturally informed NLP.

Abstract

In recent years, multilingual Large Language Models (LLMs) have become an inseparable part of daily life, making it crucial for them to master the rules of conversational language in order to communicate effectively with users. While previous work has evaluated LLMs' understanding of figurative language in high-resource languages, their performance in low-resource languages remains underexplored. In this paper, we introduce MasalBench, a comprehensive benchmark for assessing LLMs' contextual and cross-cultural understanding of Persian proverbs, which are a key component of conversation in this low-resource language. We evaluate eight state-of-the-art LLMs on MasalBench and find that they perform well in identifying Persian proverbs in context, achieving accuracies above 0.90. However, their performance drops considerably when tasked with identifying equivalent English proverbs, with the best model achieving 0.79 accuracy. Our findings highlight the limitations of current LLMs in cultural knowledge and analogical reasoning, and they provide a framework for assessing cross-cultural understanding in other low-resource languages. MasalBench is available at https://github.com/kalhorghazal/MasalBench.

Figures (7)

• Figure 1: Pipeline illustrating the construction of the benchmark for assessing LLMs' contextual and cross-cultural understanding of Persian proverbs.
• Figure 2: Heatmap of the proportion of distractor option selections for contextual understanding across LLMs.
• Figure 3: Example prompt for generating dialogues for Persian proverbs, along with its English translation.
• Figure 4: Example prompt for generating distractor options in multiple-choice questions for contextual understanding, along with its English translation.
• Figure 5: Example prompt for generating English equivalents of Persian proverbs for cross-cultural understanding, along with its English translation.