Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Kinship Data Benchmark for Multi-hop Reasoning

Tianda Sun, Dimitar Kazakov

TL;DR

KinshipQA presents a contamination-proof benchmark for evaluating multi-hop reasoning over culturally grounded kinship data. It introduces a scalable pipeline to generate large, realistic genealogies under diverse kinship rules and derives text-based questions that require implicit relational chaining and cultural disambiguation. The study reveals that both reasoning complexity and cultural variation independently limit LLM performance, with a pronounced cultural override effect when cultural rules diverge from biological defaults, and that these factors interact most strongly at 2–3 hops. This work highlights the need for architectures capable of following culture-specific rules over multi-hop chains and provides a versatile framework for evaluating reasoning in linguistically and culturally diverse settings.

Abstract

Large language models (LLMs) are increasingly evaluated on their ability to perform multi-hop reasoning, i.e., to combine multiple pieces of information into a coherent inference. We introduce KinshipQA, a benchmark designed to probe this capability through reasoning over kinship relations. The central contribution of our work is a generative pipeline that produces, on demand, large-scale, realistic, and culture-specific genealogical data: collections of interconnected family trees that satisfy explicit marriage constraints associated with different kinship systems. This allows task difficulty, cultural assumptions, and relational depth to be systematically controlled and varied. From these genealogies, we derive textual inference tasks that require reasoning over implicit relational chains. We evaluate the resulting benchmark using six state-of-the-art LLMs, spanning both open-source and closed-source models, under a uniform zero-shot protocol with deterministic decoding. Performance is measured using exact-match and set-based metrics. Our results demonstrate that KinshipQA yields a wide spread of outcomes and exposes systematic differences in multi-hop reasoning across models and cultural settings.

Kinship Data Benchmark for Multi-hop Reasoning

TL;DR

KinshipQA presents a contamination-proof benchmark for evaluating multi-hop reasoning over culturally grounded kinship data. It introduces a scalable pipeline to generate large, realistic genealogies under diverse kinship rules and derives text-based questions that require implicit relational chaining and cultural disambiguation. The study reveals that both reasoning complexity and cultural variation independently limit LLM performance, with a pronounced cultural override effect when cultural rules diverge from biological defaults, and that these factors interact most strongly at 2–3 hops. This work highlights the need for architectures capable of following culture-specific rules over multi-hop chains and provides a versatile framework for evaluating reasoning in linguistically and culturally diverse settings.

Abstract

Large language models (LLMs) are increasingly evaluated on their ability to perform multi-hop reasoning, i.e., to combine multiple pieces of information into a coherent inference. We introduce KinshipQA, a benchmark designed to probe this capability through reasoning over kinship relations. The central contribution of our work is a generative pipeline that produces, on demand, large-scale, realistic, and culture-specific genealogical data: collections of interconnected family trees that satisfy explicit marriage constraints associated with different kinship systems. This allows task difficulty, cultural assumptions, and relational depth to be systematically controlled and varied. From these genealogies, we derive textual inference tasks that require reasoning over implicit relational chains. We evaluate the resulting benchmark using six state-of-the-art LLMs, spanning both open-source and closed-source models, under a uniform zero-shot protocol with deterministic decoding. Performance is measured using exact-match and set-based metrics. Our results demonstrate that KinshipQA yields a wide spread of outcomes and exposes systematic differences in multi-hop reasoning across models and cultural settings.
Paper Structure (49 sections, 2 figures, 9 tables)

This paper contains 49 sections, 2 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi-Hop Reasoning Benchmarks
  4. Kinship Reasoning Benchmarks
  5. Cultural Knowledge in NLP
  6. Methodology
  7. Task Formulation
  8. Kinship Systems as Test Domains
  9. Benchmark Generator Pipeline
  10. Question Categories
  11. Comparing LLMs with KinshipQA
  12. Results and Evaluation
  13. Experimental Setup
  14. Models.
  15. Evaluation Protocol.
  16. ...and 34 more sections

Figures (2)

  • Figure 1: KinshipQA generation pipeline. (1) Population Simulation generates multi-generational family trees (60--70 individuals) respecting culture-specific marriage rules. (2) RDF/OWL Encoding formalizes structures using dual namespaces: family: for biological relationships and kin: for cultural classifications. (3) Question Generation creates questions across four categories with controlled n-hop complexity using path-based templates. (4) Proof Graph Extraction derives minimal reasoning subgraphs required to answer each question. (5) NL Context Serialisation converts the relevant RDF subgraph into natural language context for LLM evaluation.
  • Figure 2: Performance degradation by reasoning complexity (n-hops). Non-override systems maintain high accuracy across all hop counts, while the rest show degradation at 3-hop complexity.