Hyperphantasia: A Benchmark for Evaluating the Mental Visualization Capabilities of Multimodal LLMs

TL;DR

Hyperphantasia introduces a synthetic benchmark to assess the elusive skill of mental visualization in multimodal large language models. The dataset comprises four puzzle types split into Interpolation and Extrapolation, with three difficulty levels and 1200 samples, designed to probe internal construction, interpolation, and extrapolation of visual patterns. Across a broad set of state-of-the-art models, results reveal a substantial gap between human and machine mental-visualization capabilities, with task-dependent strengths and frequent non-grounded inferences. The study also demonstrates that reinforcement learning with diverse, moderately challenging data can improve generalization to harder and novel tasks, and it highlights robustness issues when models encounter slight visual perturbations. Hyperphantasia thus serves as a targeted benchmark to drive research toward robust internal visual thinking and grounded visual reasoning.

Abstract

Mental visualization, the ability to construct and manipulate visual representations internally, is a core component of human cognition and plays a vital role in tasks involving reasoning, prediction, and abstraction. Despite the rapid progress of Multimodal Large Language Models (MLLMs), current benchmarks primarily assess passive visual perception, offering limited insight into the more active capability of internally constructing visual patterns to support problem solving. Yet mental visualization is a critical cognitive skill in humans, supporting abilities such as spatial navigation, predicting physical trajectories, and solving complex visual problems through imaginative simulation. To bridge this gap, we introduce Hyperphantasia, a synthetic benchmark designed to evaluate the mental visualization abilities of MLLMs through four carefully constructed puzzles. Each puzzle is procedurally generated and presented at three difficulty levels, enabling controlled analysis of model performance across increasing complexity. Our comprehensive evaluation of state-of-the-art models reveals a substantial gap between the performance of humans and MLLMs. Additionally, we explore the potential of reinforcement learning to improve visual simulation capabilities. Our findings suggest that while some models exhibit partial competence in recognizing visual patterns, robust mental visualization remains an open challenge for current MLLMs.

Figures (5)

• Figure 1: Overview of puzzles in Hyperphantasia
• Figure 2: Examples of Hyperphantasia puzzles across difficulty levels. (a) In Connect the Dots, models must identify an object among four options based on samples of its outer contour. (b) In Linear Trajectory, the model must mentally complete a missing edge of a triangle and identify the correct vertex. (c) In Parabolic Trajectory, the model is asked to extrapolate the partial trajectory of a dropped ball and predict its impact point. We omit Seven Segments as it consists of uniformly styled dot grids and differs only in the number of digits per puzzle.
• Figure 3: An example of Seven Segments puzzle with added edges as visual cues.
• Figure 4: Images which caused hallucination in the models.
• Figure 5: Extrapolation images shown to the models.