Cube Bench: A Benchmark for Spatial Visual Reasoning in MLLMs

TL;DR

Cube Bench introduces a compact, simulator-based benchmark using the Rubik’s Cube to study the full see→evaluate→act→reflect→recover loop in multimodal LLMs under complete state information. It decomposes performance into seven targeted tests, each probing distinct stages of sequential spatial reasoning, with deterministic episode generation and an exact distance-to-solved metric to isolate failure modes. Across seven models, results show robust perception does not guarantee long-horizon control: performance collapses with depth, post-error recovery is rare, and even top models struggle without explicit mechanisms for pre-action evaluation and recovery. Reflection helps only under controlled (labelled or unredacted) settings, and large gaps remain between frontier and open-weight models, highlighting the need for better planning and recovery mechanisms. Cube Bench offers a reproducible framework to diagnose and compare future approaches to sequential spatial reasoning in MLLMs.

Abstract

We introduce Cube Bench, a Rubik's-cube benchmark for evaluating spatial and sequential reasoning in multimodal large language models (MLLMs). The benchmark decomposes performance into five skills: (i) reconstructing cube faces from images and text, (ii) choosing the optimal next move, (iii) predicting the outcome of a candidate move without applying it, (iv) executing multi-step plans while recovering from mistakes, and (v) detecting and revising one's own errors. Using a shared set of scrambled cube states, identical prompts and parsers, and a single distance-to-solved metric, we compare recent MLLMs side by side as a function of scramble depth. Across seven MLLMs, accuracy drops sharply with depth; once a trajectory stalls or diverges, models rarely recover, and high face-reconstruction accuracy does not guarantee competent action selection or multi-step execution. A pronounced closed- vs open-source gap emerges: the strongest closed model leads on both single-step perception tasks and multi-step control tasks, while open-weight models cluster near chance on the hardest settings; yet even the best MLLM degrades at higher cube complexity. A simple self-correction via reflective thinking yields modest gains but can also introduce overthinking. Cube Bench offers a compact, reproducible probe of sequential spatial reasoning in MLLMs.

Figures (5)

• Figure 1: Cube Bench flow: Given depth and seed, a VirtualCube state yields (i) a rendered image, (ii) a canonical text state, and (iii) a four-option set (A–D). The MLLM consumes the modalities and prompt, and returns output: Yes|No (Verification), A|B|C|D (Move Prediction / Closed-Loop), or DECREASE|NO_CHANGE|INCREASE (Causal Move-Effect).
• Figure 2: Step-by-step closed-loop results. TA% is Teacher-Adherence, i.e., $\frac{\text{avg correct steps}}{d}\times 100$. Perfect% is the share of episodes with perfect teacher adherence. Accuracies use unconditional denominators (episodes that ended early count as incorrect at later steps). Bands (w.r.t TA%): 0--20% 20--60% 60--100%.
• Figure 3: Move-Effect $\kappa$ vs. Closed-Loop TA% at depth $d{=}1$. Pearson $r=0.997$ (95% CI [0.972, 1.000]); least-squares fit shown. The relation weakens among models with $\kappa\!\approx\!0$.
• Figure 4: Example Rubik's Cube States. The left image shows a cube in its fully solved configuration, where each face consists of a single color. The right image displays a cube in a scrambled state, typical of a starting position in a puzzle-solving task.
• Figure 5: Correlation between Pre-action Causal Evaluation ($\kappa$) and Closed-Loop Teacher Adherence (TA%) across depths. At $d=1$, the association is strong ($r=0.997$), showing that models capable of causal forecasting perform better at control. However, as depth increases ($d=2, 3$), the relation weakens as open-weight models cluster near chance performance while the frontier model degrades.