Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

No Shortcuts to Culture: Indonesian Multi-hop Question Answering for Complex Cultural Understanding

Vynska Amalia Permadi, Xingwei Tan, Nafise Sadat Moosavi, Nikos Aletras

TL;DR

This work introduces ID-MoCQA, the first large-scale Indonesian multi-hop cultural QA benchmark designed to assess deep cultural reasoning beyond single facts. It presents a two-stage framework that automatically expands single-hop IndoCulture questions into multi-hop items across six clue types and validates them via a multi-stage human–LLM pipeline in both Indonesian and English. Across frontier LLMs, open multilingual models, and region-specific Indonesian models, the study shows persistent gaps in cultural multi-hop reasoning, with larger models and language familiarity yielding advantages in Indonesian. The results underscore the need for targeted methods to improve culturally aware reasoning in LLMs, and ID-MoCQA provides a resource for evaluating and driving progress in multilingual, culturally grounded QA systems.

Abstract

Understanding culture requires reasoning across context, tradition, and implicit social knowledge, far beyond recalling isolated facts. Yet most culturally focused question answering (QA) benchmarks rely on single-hop questions, which may allow models to exploit shallow cues rather than demonstrate genuine cultural reasoning. In this work, we introduce ID-MoCQA, the first large-scale multi-hop QA dataset for assessing the cultural understanding of large language models (LLMs), grounded in Indonesian traditions and available in both English and Indonesian. We present a new framework that systematically transforms single-hop cultural questions into multi-hop reasoning chains spanning six clue types (e.g., commonsense, temporal, geographical). Our multi-stage validation pipeline, combining expert review and LLM-as-a-judge filtering, ensures high-quality question-answer pairs. Our evaluation across state-of-the-art models reveals substantial gaps in cultural reasoning, particularly in tasks requiring nuanced inference. ID-MoCQA provides a challenging and essential benchmark for advancing the cultural competency of LLMs.

No Shortcuts to Culture: Indonesian Multi-hop Question Answering for Complex Cultural Understanding

TL;DR

This work introduces ID-MoCQA, the first large-scale Indonesian multi-hop cultural QA benchmark designed to assess deep cultural reasoning beyond single facts. It presents a two-stage framework that automatically expands single-hop IndoCulture questions into multi-hop items across six clue types and validates them via a multi-stage human–LLM pipeline in both Indonesian and English. Across frontier LLMs, open multilingual models, and region-specific Indonesian models, the study shows persistent gaps in cultural multi-hop reasoning, with larger models and language familiarity yielding advantages in Indonesian. The results underscore the need for targeted methods to improve culturally aware reasoning in LLMs, and ID-MoCQA provides a resource for evaluating and driving progress in multilingual, culturally grounded QA systems.

Abstract

Understanding culture requires reasoning across context, tradition, and implicit social knowledge, far beyond recalling isolated facts. Yet most culturally focused question answering (QA) benchmarks rely on single-hop questions, which may allow models to exploit shallow cues rather than demonstrate genuine cultural reasoning. In this work, we introduce ID-MoCQA, the first large-scale multi-hop QA dataset for assessing the cultural understanding of large language models (LLMs), grounded in Indonesian traditions and available in both English and Indonesian. We present a new framework that systematically transforms single-hop cultural questions into multi-hop reasoning chains spanning six clue types (e.g., commonsense, temporal, geographical). Our multi-stage validation pipeline, combining expert review and LLM-as-a-judge filtering, ensures high-quality question-answer pairs. Our evaluation across state-of-the-art models reveals substantial gaps in cultural reasoning, particularly in tasks requiring nuanced inference. ID-MoCQA provides a challenging and essential benchmark for advancing the cultural competency of LLMs.
Paper Structure (70 sections, 4 figures, 12 tables)

This paper contains 70 sections, 4 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Cultural Competence in Sociolinguistics
  4. Cultural QA Benchmarks
  5. Automatic QA Data Construction
  6. Multi-hop QA Generation Framework
  7. Base Single-hop Question Collection
  8. From Single-hop to Multi-hop
  9. First-hop question type (clue type).
  10. Bilingual multi-hop question generation.
  11. Dataset Validation
  12. Initial Quality Assessment
  13. LLM-as-a-Judge
  14. Multi-Annotator Validation.
  15. Inter-annotator Agreement.
  16. ...and 55 more sections

Figures (4)

  • Figure 1: Single to multi-hop transformation from IndoCulture koto-etal-2024-indoculture to ID-MoCQA. Left: Original question about fabric souvenirs with origin province. Right: Our expansion requires first predicting the province (North Sumatra) through cultural clues (Tor-tor dance), then answering the question.
  • Figure 2: ID-MoCQA dataset creation pipeline. Left (Automatic QA Expansion): (1) Collection of province-specific questions from IndoCulture; (2) Expansion to multi-hop questions using Claude-3.7-Sonnet with varied clue types, generating bilingual (Indonesian/English) versions. Right (Dataset Validation): (3) Human assessment of factuality, clarity, and cultural accuracy; (4) Quality verification via LLM-as-a-judge; (5) Multi-hop verification to check quality, ensure language balance, and assess naturalness and difficulty.
  • Figure 3: Breakdown of model predictions (%) by first-hop (province-level) and second-hop (final answer) correctness for English and Indonesian.
  • Figure 4: Improvement (%) from CoT over Non-CoT prompting across models and question types.