Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Your Reasoning Benchmark May Not Test Reasoning: Revealing Perception Bottleneck in Abstract Reasoning Benchmarks

Xinhe Wang, Jin Huang, Xingjian Zhang, Tianhao Wang, Jiaqi W. Ma

TL;DR

This work challenges the view that ARC-style benchmarks probe core reasoning by showing that visual perception largely constrains performance. It introduces a two-stage pipeline that first converts images to natural-language descriptions and then applies reasoning on those descriptions, revealing substantial gains when perception is enhanced. Across Mini-ARC, ACRE, and Bongard-LOGO, approximately 80% of failures arise from perceptual errors, suggesting that ARC-style tasks conflate perception with inductive reasoning. The findings advocate evaluation protocols that separate perception from reasoning to accurately measure progress toward true fluid intelligence.

Abstract

Reasoning benchmarks such as the Abstraction and Reasoning Corpus (ARC) and ARC-AGI are widely used to assess progress in artificial intelligence and are often interpreted as probes of core, so-called ``fluid'' reasoning abilities. Despite their apparent simplicity for humans, these tasks remain challenging for frontier vision-language models (VLMs), a gap commonly attributed to deficiencies in machine reasoning. We challenge this interpretation and hypothesize that the gap arises primarily from limitations in visual perception rather than from shortcomings in inductive reasoning. To verify this hypothesis, we introduce a two-stage experimental pipeline that explicitly separates perception and reasoning. In the perception stage, each image is independently converted into a natural-language description, while in the reasoning stage a model induces and applies rules using these descriptions. This design prevents leakage of cross-image inductive signals and isolates reasoning from perception bottlenecks. Across three ARC-style datasets, Mini-ARC, ACRE, and Bongard-LOGO, we show that the perception capability is the dominant factor underlying the observed performance gap by comparing the two-stage pipeline with against standard end-to-end one-stage evaluation. Manual inspection of reasoning traces in the VLM outputs further reveals that approximately 80 percent of model failures stem from perception errors. Together, these results demonstrate that ARC-style benchmarks conflate perceptual and reasoning challenges and that observed performance gaps may overstate deficiencies in machine reasoning. Our findings underscore the need for evaluation protocols that disentangle perception from reasoning when assessing progress in machine intelligence.

Your Reasoning Benchmark May Not Test Reasoning: Revealing Perception Bottleneck in Abstract Reasoning Benchmarks

TL;DR

This work challenges the view that ARC-style benchmarks probe core reasoning by showing that visual perception largely constrains performance. It introduces a two-stage pipeline that first converts images to natural-language descriptions and then applies reasoning on those descriptions, revealing substantial gains when perception is enhanced. Across Mini-ARC, ACRE, and Bongard-LOGO, approximately 80% of failures arise from perceptual errors, suggesting that ARC-style tasks conflate perception with inductive reasoning. The findings advocate evaluation protocols that separate perception from reasoning to accurately measure progress toward true fluid intelligence.

Abstract

Reasoning benchmarks such as the Abstraction and Reasoning Corpus (ARC) and ARC-AGI are widely used to assess progress in artificial intelligence and are often interpreted as probes of core, so-called ``fluid'' reasoning abilities. Despite their apparent simplicity for humans, these tasks remain challenging for frontier vision-language models (VLMs), a gap commonly attributed to deficiencies in machine reasoning. We challenge this interpretation and hypothesize that the gap arises primarily from limitations in visual perception rather than from shortcomings in inductive reasoning. To verify this hypothesis, we introduce a two-stage experimental pipeline that explicitly separates perception and reasoning. In the perception stage, each image is independently converted into a natural-language description, while in the reasoning stage a model induces and applies rules using these descriptions. This design prevents leakage of cross-image inductive signals and isolates reasoning from perception bottlenecks. Across three ARC-style datasets, Mini-ARC, ACRE, and Bongard-LOGO, we show that the perception capability is the dominant factor underlying the observed performance gap by comparing the two-stage pipeline with against standard end-to-end one-stage evaluation. Manual inspection of reasoning traces in the VLM outputs further reveals that approximately 80 percent of model failures stem from perception errors. Together, these results demonstrate that ARC-style benchmarks conflate perceptual and reasoning challenges and that observed performance gaps may overstate deficiencies in machine reasoning. Our findings underscore the need for evaluation protocols that disentangle perception from reasoning when assessing progress in machine intelligence.
Paper Structure (28 sections, 2 equations, 5 figures, 9 tables)

This paper contains 28 sections, 2 equations, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reasoning Benchmarks.
  4. State-of-the-Art Performance on the ARC Benchmarks.
  5. Limitations of the ARC Benchmarks.
  6. Verifying the Hypothesis with a Two-Stage Pipeline
  7. Abstract Reasoning Tasks
  8. Quantifying the Perception Bottleneck with a Two-Stage Pipeline
  9. Design Principles of the Perception Stage.
  10. Formal Description of the Two-Stage Pipeline.
  11. Two Evaluation Settings
  12. Setting 1: Same-Model Perception.
  13. Setting 2: Stronger-Model Perception.
  14. Fine-Grained Error Attribution with Four Categories
  15. Conceptual Decomposition of the Task-Solving Process.
  16. ...and 13 more sections

Figures (5)

  • Figure 1: An example ARC problem shown in 2-D visually.
  • Figure 2: The same ARC problem shown in a serialized format.
  • Figure 4: Example tasks of two ARC-style benchmarks, Mini-ARC kim2022playgrounds and Bongard-LOGO nie2020bongard. The task in Mini-ARC maps image inputs to image outputs. The task in Bongard-LOGO maps image inputs to binary outputs (positive or negative). In both cases, visual objects that are immediately recognizable by humans serve as the critical basis for the latent mapping rules.
  • Figure 5: Dependency graph illustrating the four-step task-solving process. Errors in earlier stages propagate to subsequent stages, affecting final predictions.
  • Figure 6: Visualization of how errors evolve from one configuration to another.