AmharicStoryQA: A Multicultural Story Question Answering Benchmark in Amharic

TL;DR

AmharicStoryQA addresses the gap in culturally grounded evaluation for low-resource languages by introducing a long-sequence, regionally diverse Amharic QA benchmark derived from Ethiopian folktales. The dataset includes 224 stories across nine regions, yielding 571 training and 649 test QA instances (244 stories) with both MCQA and generation tasks, created via GPT-4.1 and enhanced by human validation and SSA-COMET-based translation quality checks. Evaluations across seven open-source LLMs reveal a persistent Amharic narrative-understanding gap, pronounced regional differences, and a dissociation between MCQA and generative performance. Fine-tuning with a bilingual English–Amharic, culturally grounded dataset via LoRA improves Amharic performance and suggests cross-lingual transfer as a viable path to address observed limitations, underscoring the need for culturally aware benchmarks for low-resource languages.

Abstract

With the growing emphasis on multilingual and cultural evaluation benchmarks for large language models, language and culture are often treated as synonymous, and performance is commonly used as a proxy for a models understanding of a given language. In this work, we argue that such evaluations overlook meaningful cultural variation that exists within a single language. We address this gap by focusing on narratives from different regions of Ethiopia and demonstrate that, despite shared linguistic characteristics, region-specific and domain-specific content substantially influences language evaluation outcomes. To this end, we introduce \textbf{\textit{AmharicStoryQA}}, a long-sequence story question answering benchmark grounded in culturally diverse narratives from Amharic-speaking regions. Using this benchmark, we reveal a significant narrative understanding gap in existing LLMs, highlight pronounced regional differences in evaluation results, and show that supervised fine-tuning yields uneven improvements across regions and evaluation settings. Our findings emphasize the need for culturally grounded benchmarks that go beyond language-level evaluation to more accurately assess and improve narrative understanding in low-resource languages.

Figures (11)

• Figure 1: Overview of the AmharicStoryQA dataset creation process, from Amharic and English parallel stories through question generation, manual translation rating, quality control, and train–test split verification.
• Figure 2: Regional semantic density of Amharic stories across regions in the AmharicStoryQA corpus. Each contour shows a kernel density estimate (KDE) over 2D PCA projections of multilingual MiniLM sentence embeddings, coloured by region. The plot shows that regional stories largely share a common semantic space, with subtle outer-region contours revealing culturally specific variations within the same language.
• Figure 3: Average Stories Sequence Length by Region
• Figure 4: Inter-Rater Reliability agreement across Correctness, Linguistic, and Comprehension categories assessed with three human evaluators
• Figure 5: Regional Performance. Performance of the two differently performing models (top) (Gemma-3-27b-it) and (Llama 3.2 1B Instruct), showing performance variation across narratives from different regions in our dataset.