Artificial Phantasia: Evidence for Propositional Reasoning-Based Mental Imagery in Large Language Models

TL;DR

This study investigates whether large language models can perform mental imagery tasks through propositional reasoning, challenging the view that imagery requires pictorial representations. By designing 60 instruction sets with 48 novel items inspired by Finke 1989 and comparing a 100-person human baseline against a wide range of models, the authors assess imagery-like reasoning in language models. The strongest models (GPT-5 and the o3 family) outperform humans, suggesting emergent propositional reasoning-based capabilities for imagery-dependent tasks, while image-aided prompts often hinder performance. These findings contribute to cognitive science debates on imagery formats and establish a rigorous, out-of-distribution benchmark for evaluating advanced cognitive capacities in LLMs, with implications for future model development and benchmarking paradigms.

Abstract

This study offers a novel approach for benchmarking complex cognitive behavior in artificial systems. Almost universally, Large Language Models (LLMs) perform best on tasks which may be included in their training data and can be accomplished solely using natural language, limiting our understanding of their emergent sophisticated cognitive capacities. In this work, we created dozens of novel items of a classic mental imagery task from cognitive psychology. A task which, traditionally, cognitive psychologists have argued is solvable exclusively via visual mental imagery (i.e., language alone would be insufficient). LLMs are perfect for testing this hypothesis. First, we tested several state-of-the-art LLMs by giving text-only models written instructions and asking them to report the resulting object after performing the transformations in the aforementioned task. Then, we created a baseline by testing 100 human subjects in exactly the same task. We found that the best LLMs performed significantly above average human performance. Finally, we tested reasoning models set to different levels of reasoning and found the strongest performance when models allocate greater amounts of reasoning tokens. These results provide evidence that the best LLMs may have the capability to complete imagery-dependent tasks despite the non-pictorial nature of their architectures. Our study not only demonstrates an emergent cognitive capacity in LLMs while performing a novel task, but it also provides the field with a new task that leaves lots of room for improvement in otherwise already highly capable models. Finally, our findings reignite the debate over the formats of representation of visual imagery in humans, suggesting that propositional reasoning (or at least non-imagistic reasoning) may be sufficient to complete tasks that were long-thought to be imagery-dependent.

Figures (15)

• Figure 1: One of the instruction sets introduced in finke1989. Here, subjects are meant to recognize from the resulting mental image that the final imagined object looks like an umbrella. The instructions have been rewritten to be clearer both for prompting LLMs, as well as for human understandability.
• Figure 2: One of our new instruction sets demonstrating the slightly increased cognitive complexity and more ambiguous canonical form ("balloons", "flower bouquet", or "ice cream", among others). Note the usage of two letters in the first step, the abstract reference to existing symbols and scenes, and the final shape not being determinable until the final step.
• Figure 3: Performance results in humans and LLMs. Data shows proportion of maximum possible score for all tested models. Only GPT-5, o3, and o3-Pro significantly surpass the human baseline. Error bars indicate 95% confidence intervals.
• Figure S1: An example image generation output from GPT-image-1 in combination with o3. No seed image was given to the model, but subsequent steps retained the previous image and asked for its modification.
• Figure S2: 95% confidence intervals showing the differences between Single-Context and Multiple-Context. Single-Context has a slight, non-significant edge in most cases. All context variant comparisons were non-significant after correcting for multiple-comparisons.