MentisOculi: Revealing the Limits of Reasoning with Mental Imagery

TL;DR

MentisOculi is developed, a procedural, stratified suite of multi-step reasoning problems amenable to visual solution, tuned to challenge frontier models, suggesting that despite their inherent appeal, visual thoughts do not yet benefit model reasoning.

Abstract

Frontier models are transitioning from multimodal large language models (MLLMs) that merely ingest visual information to unified multimodal models (UMMs) capable of native interleaved generation. This shift has sparked interest in using intermediate visualizations as a reasoning aid, akin to human mental imagery. Central to this idea is the ability to form, maintain, and manipulate visual representations in a goal-oriented manner. To evaluate and probe this capability, we develop MentisOculi, a procedural, stratified suite of multi-step reasoning problems amenable to visual solution, tuned to challenge frontier models. Evaluating visual strategies ranging from latent tokens to explicit generated imagery, we find they generally fail to improve performance. Analysis of UMMs specifically exposes a critical limitation: While they possess the textual reasoning capacity to solve a task and can sometimes generate correct visuals, they suffer from compounding generation errors and fail to leverage even ground-truth visualizations. Our findings suggest that despite their inherent appeal, visual thoughts do not yet benefit model reasoning. MentisOculi establishes the necessary foundation to analyze and close this gap across diverse model families.

Figures (23)

• Figure 1: Mentis-Oculi comprises five visual reasoning tasks designed to be best-solved with mental imagery. Collectively, the tasks require models to solve multi-step reasoning problems with geometric constraints. Success hinges on the ability to maintain a visual representation with high fidelity and consistent geometry under affine transformations. Each task is procedurally generated across five difficulty levels, scaling with the number of operations required from one (left) to five (right); see \ref{['app:generation-details']} for details.
• Figure 2: MLLMs and UMMs display similar failure patterns across tasks: Performance degrades noticeably with difficulty and falls below chance at Level 5, indicating that visual reasoning limitations are task-agnostic. Data for all levels in \ref{['fig:all-tasks-all-levels']}.
• Figure 3: Different kinds of mental imagery do not greatly improve multi-step reasoning on Rush Hour: Compared to MLLMs, the latent visual reasoning modelMirage that is fine-tuned to generate interleaved visual latent tokens shows some improvement (especially considering its relatively weak base model), but with diminishing returns at harder levels. In contrast, UMMs that interleave generated images and texts generally perform below their MLLM counterparts. The video modelVeo 3.1 operates purely in pixel space and breaks down quickly as difficulty increases. * samples omitted (no answer provided)
• Figure 4: MLLMs have the competence to solve Rush Hour when prompted with a transcription of the task. Gemini 3 and GPT-5.1 even perform on par with humans, even though the text-only Rush Hour requires mathematically solving for possible collisions.
• Figure 5: UMM performance faces a dual issue: Generation errors are ubiquitous---performance on all tasks increases with oracle visualizations. However, on most tasks, UMMs fail to utilize even correct visuals to aid their reasoning, which we term interpretation errors. Data for all levels in \ref{['fig:interleaved-all-tasks-all-levels']}.