Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SokoBench: Evaluating Long-Horizon Planning and Reasoning in Large Language Models

Sebastiano Monti, Carlo Nicolini, Gianni Pellegrini, Jacopo Staiano, Bruno Lepri

TL;DR

This work probes the long-horizon planning abilities of Large Reasoning Models by using a highly controlled Sokoban corridor variant to isolate planning from state persistence. It contrasts pure reasoning (1-shot) with an LLM-Modulo setup that delegates planning to external PDDL-based solvers, revealing a pronounced drop in planning accuracy once the horizon exceeds about 25 moves. The study finds that counting drift and weak internal state representations largely drive failures, even when external tools are available, though PDDL tooling provides modest improvements. Collectively, the results suggest fundamental architectural limits in current LRMs for reliable long-horizon planning and motivate future work on explicit symbolic grounding or improved memory/visited-state mechanisms.

Abstract

Although the capabilities of large language models have been increasingly tested on complex reasoning tasks, their long-horizon planning abilities have not yet been extensively investigated. In this work, we provide a systematic assessment of the planning and long-horizon reasoning capabilities of state-of-the-art Large Reasoning Models (LRMs). We propose a novel benchmark based on Sokoban puzzles, intentionally simplified to isolate long-horizon planning from state persistence. Our findings reveal a consistent degradation in planning performance when more than 25 moves are required to reach the solution, suggesting a fundamental constraint on forward planning capacity. We show that equipping LRMs with Planning Domain Definition Language (PDDL) parsing, validation, and solving tools allows for modest improvements, suggesting inherent architectural limitations which might not be overcome by test-time scaling approaches alone.

SokoBench: Evaluating Long-Horizon Planning and Reasoning in Large Language Models

TL;DR

This work probes the long-horizon planning abilities of Large Reasoning Models by using a highly controlled Sokoban corridor variant to isolate planning from state persistence. It contrasts pure reasoning (1-shot) with an LLM-Modulo setup that delegates planning to external PDDL-based solvers, revealing a pronounced drop in planning accuracy once the horizon exceeds about 25 moves. The study finds that counting drift and weak internal state representations largely drive failures, even when external tools are available, though PDDL tooling provides modest improvements. Collectively, the results suggest fundamental architectural limits in current LRMs for reliable long-horizon planning and motivate future work on explicit symbolic grounding or improved memory/visited-state mechanisms.

Abstract

Although the capabilities of large language models have been increasingly tested on complex reasoning tasks, their long-horizon planning abilities have not yet been extensively investigated. In this work, we provide a systematic assessment of the planning and long-horizon reasoning capabilities of state-of-the-art Large Reasoning Models (LRMs). We propose a novel benchmark based on Sokoban puzzles, intentionally simplified to isolate long-horizon planning from state persistence. Our findings reveal a consistent degradation in planning performance when more than 25 moves are required to reach the solution, suggesting a fundamental constraint on forward planning capacity. We show that equipping LRMs with Planning Domain Definition Language (PDDL) parsing, validation, and solving tools allows for modest improvements, suggesting inherent architectural limitations which might not be overcome by test-time scaling approaches alone.
Paper Structure (24 sections, 3 equations, 8 figures, 3 tables)

This paper contains 24 sections, 3 equations, 8 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Benchmarks for LLM Planning
  4. Sokoban as a Benchmark for Planning
  5. Methods
  6. Sokoban game
  7. Dataset
  8. Experimental Setup
  9. 1-shot Inference
  10. LLM-Modulo
  11. Evaluation
  12. Accuracy:
  13. Prefix accuracy:
  14. Manhattan Distance:
  15. Results
  16. ...and 9 more sections

Figures (8)

  • Figure 1:
  • Figure 2: Panel (a) represents a simple schema of our LLM-modulo pipeline. The detailed input prompts are collected in Appendix \ref{['sec:agentic_system_prompt']}, while an example of output is shown in Panel (b).
  • Figure 3: Accuracy and number of reasoning tokens for LRM experiment. Left: average accuracy. Error bars are computed as the 5th and 95th percentile of responses. Right: scaling behaviour of reasoning length against corridor length filtered for correct solutions only.
  • Figure 4: Number of completion tokens produced by each model as a function of corridor length. Separate linear regressions are fitted for correct and incorrect responses, with outliers excluded. A small jitter is added to the x-axis to improve visualization.
  • Figure 5: Other useful metrics represented as functions of the corridor's length. Panel (a) represents prefix accuracy, computed as described in Equation \ref{['eq:prefix_accuracy']}. Panel (b) represents Manhattan distance, computed as in Equation \ref{['eq:manhattan_distance']}. Models' colors are the same as in Figure \ref{['fig:three_plots']}.
  • ...and 3 more figures