Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Pixels to BFS: High Maze Accuracy Does Not Imply Visual Planning

Alberto G. Rodriguez Salgado

Abstract

How do multimodal models solve visual spatial tasks -- through genuine planning, or through brute-force search in token space? We introduce \textsc{MazeBench}, a benchmark of 110 procedurally generated maze images across nine controlled groups, and evaluate 16 model configurations from OpenAI, Anthropic, Google, and Alibaba. GPT-5.4 solves 91\% and Gemini 3.1 Pro 79\%, but these scores are misleading: models typically translate images into text grids and then enumerate paths step by step, consuming 1,710--22,818 tokens per solve for a task humans do quickly. Without added reasoning budgets, all configurations score only 2--12\%; on 20$\times$20 ultra-hard mazes, they hit token limits and fail. Qualitative traces reveal a common two-stage strategy: image-to-grid translation followed by token-level search, effectively BFS in prose. A text-grid ablation shows Claude Sonnet 4.6 rising from 6\% on images to 80\% when given the correct grid, isolating weak visual extraction from downstream search. When explicitly instructed not to construct a grid or perform graph search, models still revert to the same enumeration strategy. \textsc{MazeBench} therefore shows that high accuracy on visual planning tasks does not imply human-like spatial understanding.

From Pixels to BFS: High Maze Accuracy Does Not Imply Visual Planning

Abstract

How do multimodal models solve visual spatial tasks -- through genuine planning, or through brute-force search in token space? We introduce \textsc{MazeBench}, a benchmark of 110 procedurally generated maze images across nine controlled groups, and evaluate 16 model configurations from OpenAI, Anthropic, Google, and Alibaba. GPT-5.4 solves 91\% and Gemini 3.1 Pro 79\%, but these scores are misleading: models typically translate images into text grids and then enumerate paths step by step, consuming 1,710--22,818 tokens per solve for a task humans do quickly. Without added reasoning budgets, all configurations score only 2--12\%; on 2020 ultra-hard mazes, they hit token limits and fail. Qualitative traces reveal a common two-stage strategy: image-to-grid translation followed by token-level search, effectively BFS in prose. A text-grid ablation shows Claude Sonnet 4.6 rising from 6\% on images to 80\% when given the correct grid, isolating weak visual extraction from downstream search. When explicitly instructed not to construct a grid or perform graph search, models still revert to the same enumeration strategy. \textsc{MazeBench} therefore shows that high accuracy on visual planning tasks does not imply human-like spatial understanding.

Paper Structure

This paper contains 35 sections, 9 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Controlled difficulty via procedural generation.
  3. Reasoning effort as a controlled variable.
  4. Token efficiency as a window into strategy.
  5. Related Work
  6. Visual perception gaps in MLLMs.
  7. Multimodal reasoning benchmarks.
  8. Spatial reasoning in MLLMs.
  9. Test-time compute scaling.
  10. Benchmark Design
  11. Task Formulation
  12. Procedural Maze Generation
  13. Dataset Structure
  14. Experimental Setup
  15. Models.
  16. ...and 20 more sections

Figures (9)

  • Figure 2: Group A: Diagnostic (8 mazes). Empty or near-empty grids with trivial straight-line paths.
  • Figure 3: Group B: Grid Scale (15 mazes). Constant 25% wall density, grid sizes from $5 \times 5$ to $13 \times 13$.
  • Figure 4: Group C: Wall Density (15 mazes). Constant $9 \times 9$ grid, density from 0% to 45%.
  • Figure 5: Group D: Trap Ablation (12 mazes). Six matched pairs---control (no traps) and treatment (with traps)---sharing the same random seed.
  • Figure 6: Group E: Unreachable (14 mazes). All mazes have no valid path from start to goal.
  • ...and 4 more figures