CubeBench: Diagnosing Interactive, Long-Horizon Spatial Reasoning Under Partial Observations

TL;DR

CubeBench provides a minimal, verifiable diagnostic for three intertwined cognitive faculties essential to physical-world AI: 3D spatial reasoning, long-horizon state tracking, and exploration under partial observation. By framing the Rubik's Cube as a three-tier POMDP and evaluating a range of LLM-driven agents—both unaided and with tools—the paper shows a striking gap: a uniform $0.00$ pass rate on long-horizon tasks and limited symbolic proficiency even on short-horizon problems. Introducing dense rewards and external solvers isolates distinct bottlenecks, revealing that long-horizon planning and belief-state maintenance are primary weaknesses, while solver tools can mitigate but not fully resolve these deficits. The work argues for a phase-wise diagnostic approach to guide the development of physically-grounded, spatially aware agents and suggests that integrating robust external planning tools, along with advances in internal spatial models, is crucial for real-world deployment.

Abstract

Large Language Model (LLM) agents, while proficient in the digital realm, face a significant gap in physical-world deployment due to the challenge of forming and maintaining a robust spatial mental model. We identify three core cognitive challenges hindering this transition: spatial reasoning, long-horizon state tracking via mental simulation, and active exploration under partial observation. To isolate and evaluate these faculties, we introduce CubeBench, a novel generative benchmark centered on the Rubik's Cube. CubeBench uses a three-tiered diagnostic framework that progressively assesses agent capabilities, from foundational state tracking with full symbolic information to active exploration with only partial visual data. Our experiments on leading LLMs reveal critical limitations, including a uniform 0.00% pass rate on all long-horizon tasks, exposing a fundamental failure in long-term planning. We also propose a diagnostic framework to isolate these cognitive bottlenecks by providing external solver tools. By analyzing the failure modes, we provide key insights to guide the development of more physically-grounded intelligent agents.

Figures (8)

• Figure 1: An overview of the performance of leading LLMs on the CubeBench benchmark, broken down by its three diagnostic tiers. Tier 1 (Full Symbolic State) tests foundational state tracking using complete symbolic information, where the best average pass rate is only 37.5%. Tier 2 (Full Visual State) challenges visual and spatial reasoning by requiring agents to interpret a 2D unfolded map, and Tier 3 (Partial Visual State) evaluates active exploration from partial views. Across all tiers, GPT-5 emerges as the top-performing model, though the results highlight a significant performance gap between symbolic and visual reasoning tasks.
• Figure 2: Visualization of the three core cognitive challenges required for spatial reasoning.
• Figure 3: Illustration on the three-tiered task of CubeBench. Tier 1 (Full Symbolic State) provides the agent with complete state information in a string format, which makes the problem a fully observable MDP. Tier 2 (Full Visual State) presents the full state as a 2D unfolded map, which challenges the agent's visual thinking. Tier 3 (Partial Visual State) provides only a partial view of the cube (Face view or Vertex view), which requires the agent to explore the environment to gather the full state information.
• Figure 4: Illustration on the interaction protocol.
• Figure 5: Visualization of our three-part diagnostic framework for systematically evaluating LLM agents. To answer Q1, we test a basic agent with only fundamental interaction tools to establish its baseline capabilities from first principles. For Q2, we augment the agent with various dense reward signals to determine if external feedback can effectively guide its search process. Finally, to address Q3, we deploy agents with different levels of tool support to diagnose whether failures originate from high-level planning, state reconstruction, or procedural data transformation.