Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unmasking the Factual-Conceptual Gap in Persian Language Models

Alireza Sakhaeirad, Ali Ma'manpoosh, Arshia Hemmat

TL;DR

DivanBench is introduced, a diagnostic benchmark focused on superstitions and customs, arbitrary, context-dependent rules that resist simple logical deduction that demonstrate that cultural competence requires more than scaling monolingual data.

Abstract

While emerging Persian NLP benchmarks have expanded into pragmatics and politeness, they rarely distinguish between memorized cultural facts and the ability to reason about implicit social norms. We introduce DivanBench, a diagnostic benchmark focused on superstitions and customs, arbitrary, context-dependent rules that resist simple logical deduction. Through 315 questions across three task types (factual retrieval, paired scenario verification, and situational reasoning), we evaluate seven Persian LLMs and reveal three critical failures: most models exhibit severe acquiescence bias, correctly identifying appropriate behaviors but failing to reject clear violations; continuous Persian pretraining amplifies this bias rather than improving reasoning, often degrading the model's ability to discern contradictions; and all models show a 21\% performance gap between retrieving factual knowledge and applying it in scenarios. These findings demonstrate that cultural competence requires more than scaling monolingual data, as current models learn to mimic cultural patterns without internalizing the underlying schemas.

Unmasking the Factual-Conceptual Gap in Persian Language Models

TL;DR

DivanBench is introduced, a diagnostic benchmark focused on superstitions and customs, arbitrary, context-dependent rules that resist simple logical deduction that demonstrate that cultural competence requires more than scaling monolingual data.

Abstract

While emerging Persian NLP benchmarks have expanded into pragmatics and politeness, they rarely distinguish between memorized cultural facts and the ability to reason about implicit social norms. We introduce DivanBench, a diagnostic benchmark focused on superstitions and customs, arbitrary, context-dependent rules that resist simple logical deduction. Through 315 questions across three task types (factual retrieval, paired scenario verification, and situational reasoning), we evaluate seven Persian LLMs and reveal three critical failures: most models exhibit severe acquiescence bias, correctly identifying appropriate behaviors but failing to reject clear violations; continuous Persian pretraining amplifies this bias rather than improving reasoning, often degrading the model's ability to discern contradictions; and all models show a 21\% performance gap between retrieving factual knowledge and applying it in scenarios. These findings demonstrate that cultural competence requires more than scaling monolingual data, as current models learn to mimic cultural patterns without internalizing the underlying schemas.
Paper Structure (44 sections, 3 figures, 5 tables)

This paper contains 44 sections, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Motivation: Why Superstitions and Customs?
  3. Research Questions, Contributions, and Key Findings
  4. Related Work
  5. Persian LLM Evaluation
  6. Cultural Competence in LLMs
  7. Positioning Our Work
  8. Benchmark Design and Data Collection
  9. Design Principles
  10. Factual Multiple-Choice Questions
  11. Binary Belief Verification
  12. Scenario-Based Multiple-Choice Questions
  13. Author-Generated Content and Quality Control
  14. Cultural Concept Coverage
  15. Experiments
  16. ...and 29 more sections

Figures (3)

  • Figure 1: Sample Persian cultural concepts from the benchmark, spanning superstitions, traditions, and taboos.
  • Figure 2: acquiescence bias across models. Most models accept true cultural statements readily (Blue) but fail to reject false ones (Red), indicating pattern-matching rather than reasoning.
  • Figure 3: Factual Knowledge vs. Cultural Reasoning Gap. Gemma3-12B (largest bubble, top-right) achieves highest factual accuracy but shows inconsistent scenario reasoning. All models fall below the diagonal, indicating systematic difficulty in transferring factual knowledge to cultural schema application. Bubble size represents model parameters.