Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

I-RAVEN-X: Benchmarking Generalization and Robustness of Analogical and Mathematical Reasoning in Large Language and Reasoning Models

Giacomo Camposampiero, Michael Hersche, Roger Wattenhofer, Abu Sebastian, Abbas Rahimi

TL;DR

I-RAVEN-X is a symbolic benchmark designed to stress-test generalization and robustness in analogical and mathematical reasoning for large language and reasoning models. It extends the prior I-RAVEN with larger operand counts, broader attribute ranges, and simulated perceptual uncertainty across four axes (Productivity, Systematicity, confounder robustness, and distribution smoothing), while remaining fully symbolic. Empirical results show that large reasoning models (LRMs) outperform language models (LLMs) on longer reasoning chains and wider attribute ranges, and can do so with less prompt engineering, particularly in arithmetic tasks; however, LRMs remain significantly challenged by reasoning under uncertainty, struggling to explore multiple probabilistic outcomes simultaneously. The study provides a principled framework and dataset for evaluating end-to-end reasoning under perceptual uncertainty, with code and data publicly available for broader reuse and extension.

Abstract

We introduce I-RAVEN-X, a symbolic benchmark designed to evaluate generalization and robustness in analogical and mathematical reasoning for Large Language Models (LLMs) and Large Reasoning Models (LRMs). I-RAVEN-X extends I-RAVEN by increasing operand complexity, attribute range, and introducing perceptual uncertainty. Compared to LLMs, empirical results show that LRMs achieve improved productivity and systematicity on longer reasoning relations and wider attribute ranges, respectively. However, LRMs are still significantly challenged by reasoning under uncertainty and cannot effectively explore multiple probabilistic outcomes.

I-RAVEN-X: Benchmarking Generalization and Robustness of Analogical and Mathematical Reasoning in Large Language and Reasoning Models

TL;DR

I-RAVEN-X is a symbolic benchmark designed to stress-test generalization and robustness in analogical and mathematical reasoning for large language and reasoning models. It extends the prior I-RAVEN with larger operand counts, broader attribute ranges, and simulated perceptual uncertainty across four axes (Productivity, Systematicity, confounder robustness, and distribution smoothing), while remaining fully symbolic. Empirical results show that large reasoning models (LRMs) outperform language models (LLMs) on longer reasoning chains and wider attribute ranges, and can do so with less prompt engineering, particularly in arithmetic tasks; however, LRMs remain significantly challenged by reasoning under uncertainty, struggling to explore multiple probabilistic outcomes simultaneously. The study provides a principled framework and dataset for evaluating end-to-end reasoning under perceptual uncertainty, with code and data publicly available for broader reuse and extension.

Abstract

We introduce I-RAVEN-X, a symbolic benchmark designed to evaluate generalization and robustness in analogical and mathematical reasoning for Large Language Models (LLMs) and Large Reasoning Models (LRMs). I-RAVEN-X extends I-RAVEN by increasing operand complexity, attribute range, and introducing perceptual uncertainty. Compared to LLMs, empirical results show that LRMs achieve improved productivity and systematicity on longer reasoning relations and wider attribute ranges, respectively. However, LRMs are still significantly challenged by reasoning under uncertainty and cannot effectively explore multiple probabilistic outcomes.
Paper Structure (17 sections, 2 figures, 7 tables)

This paper contains 17 sections, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. I-RAVEN-X: testing generalization and robustness of reasoning in LLMs and LRMs
  4. Models and prompting techniques
  5. Results
  6. LRMs are stronger analogical and mathematical reasoners than LLMs
  7. LRMs are significantly challenged by reasoning under uncertainty
  8. Conclusion
  9. I-RAVEN dataset
  10. I-RAVEN-X implementation details
  11. Productivity and Systematicity
  12. Confounding attributes
  13. Smooth attribute values' distributions
  14. Models and prompting details
  15. LLMs
  16. ...and 2 more sections

Figures (2)

  • Figure 1: This figure highlights all the different axes of generalization and robustness to uncertainty, which I-RAVEN-X stresses. Compared to standard I-RAVEN (a), I-RAVEN-X (b) involves more panels per row (10 vs. 3) and larger attribute dynamic ranges (up to $100\times$ more values per attribute). In addition, it is possible to introduce uncertainty in the reasoning process through confounders (such as panels' background and color patterns within objects) and smoothening the attribute values' distributions (displayed on the right for the panel in position $(1,10)$) (c). We adopt a visual representation of the panels and their attributes for clarity of explanation; in practice, however, our dataset is purely symbolic and has not been extended yet to the visual domain.
  • Figure 2: RPM example from I-RAVEN.