Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When No Paths Lead to Rome: Benchmarking Systematic Neural Relational Reasoning

Anirban Das, Irtaza Khalid, Rafael Peñaloza, Steven Schockaert

TL;DR

NoRA introduces a benchmark for systematic neural relational reasoning that goes beyond path-based inference, incorporating rich, multi-relational graphs with ambiguous facts. The dataset is ASP-based, with world rules that induce answer sets; problem difficulty is quantified via depth, width, BL, and OPEC to encourage compositional generalization. Experiments show current state-of-the-art models, especially path-based and some GNNs, struggle with off-path reasoning and ambiguity, while Edge Transformers perform best but still falter on hardest splits. Large reasoning models reveal that even with explicit world rules, off-path reasoning remains challenging, highlighting a need for architectures and training paradigms capable of true non-path, multi-relational reasoning and robust rule induction. The NoRA family, including NoRA v1.1 and HetioNet, provides a broader testbed to drive progress in neural systematic reasoning and beyond-path generalization.

Abstract

Designing models that can learn to reason in a systematic way is an important and long-standing challenge. In recent years, a wide range of solutions have been proposed for the specific case of systematic relational reasoning, including Neuro-Symbolic approaches, variants of the Transformer architecture, and specialised Graph Neural Networks. However, existing benchmarks for systematic relational reasoning focus on an overly simplified setting, based on the assumption that reasoning can be reduced to composing relational paths. In fact, this assumption is hard-baked into the architecture of several recent models, leading to approaches that can perform well on existing benchmarks but are difficult to generalise to other settings. To support further progress in the field of systematic relational reasoning with neural networks, we introduce NoRA, a new benchmark which adds several levels of difficulty and requires models to go beyond path-based reasoning.

When No Paths Lead to Rome: Benchmarking Systematic Neural Relational Reasoning

TL;DR

NoRA introduces a benchmark for systematic neural relational reasoning that goes beyond path-based inference, incorporating rich, multi-relational graphs with ambiguous facts. The dataset is ASP-based, with world rules that induce answer sets; problem difficulty is quantified via depth, width, BL, and OPEC to encourage compositional generalization. Experiments show current state-of-the-art models, especially path-based and some GNNs, struggle with off-path reasoning and ambiguity, while Edge Transformers perform best but still falter on hardest splits. Large reasoning models reveal that even with explicit world rules, off-path reasoning remains challenging, highlighting a need for architectures and training paradigms capable of true non-path, multi-relational reasoning and robust rule induction. The NoRA family, including NoRA v1.1 and HetioNet, provides a broader testbed to drive progress in neural systematic reasoning and beyond-path generalization.

Abstract

Designing models that can learn to reason in a systematic way is an important and long-standing challenge. In recent years, a wide range of solutions have been proposed for the specific case of systematic relational reasoning, including Neuro-Symbolic approaches, variants of the Transformer architecture, and specialised Graph Neural Networks. However, existing benchmarks for systematic relational reasoning focus on an overly simplified setting, based on the assumption that reasoning can be reduced to composing relational paths. In fact, this assumption is hard-baked into the architecture of several recent models, leading to approaches that can perform well on existing benchmarks but are difficult to generalise to other settings. To support further progress in the field of systematic relational reasoning with neural networks, we introduce NoRA, a new benchmark which adds several levels of difficulty and requires models to go beyond path-based reasoning.

Paper Structure

This paper contains 64 sections, 18 equations, 19 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Problem setting
  4. Stories
  5. World rules
  6. Answer sets
  7. Problem formulation
  8. Example
  9. Dataset construction
  10. Data generation process
  11. Measuring problem difficulty
  12. Reasoning depth
  13. Reasoning width
  14. Non-path reasoning
  15. Training distribution and held-out test sets
  16. ...and 49 more sections

Figures (19)

  • Figure 1: Example where path-based reasoning fails: to derive that ann is todd's maternal aunt, one must consider wes, who is not on any connecting path between ann and todd.
  • Figure 2: Source entities are sam, ty1, and sam2, while target entities are joe, joe1, and joe2 for the queries accompanying stories (i), (ii) and (iii), respectively. Solid edges represent the relationships explicitly in the story. Dashed edges are entailed relationships between source–target pairs. Pink edges indicate edges that do not lie on any path between the source and target. Panel (iv) illustrates a derivation of the entailed fact in story (iii). It uses all four off-path edges, hence the query from story (iii) has an OPEC value of 4. The queries in stories (i) and (ii) each have an OPEC value of 2.
  • Figure 3: Analysis of the performance of ETs on various splits of the dataset.
  • Figure 4: (a) Breakdown of the performance of edge transformers on Test-D; (b) analysis of o3 on non-ambiguous stories; (c) a comparison between o3 and o4-mini on non-ambiguous stories.
  • Figure 5: Results for the expanded version of NoRA (v1.1) that uses recursive subgraph expansion to generate harder splits along the axes: (a) OPEC, (b) Reasoning Depth (c) BL.
  • ...and 14 more figures

Theorems & Definitions (1)

  • Definition 1: Hard Ambiguous Problem Instances