Autonomous Imagination: Closed-Loop Decomposition of Visual-to-Textual Conversion in Visual Reasoning for Multimodal Large Language Models

TL;DR

This paper tackles the visual-to-textual conversion bottleneck in multimodal LLMs by introducing autonomous imagination, a closed-loop framework where the model iteratively modifies visual inputs within an imagination space to decompose complex visual tasks. The approach defines a minimal operator set (focus, ignore, transform) and supports both 2D images and 3D Gaussian representations, enabling training-free, progressive simplification of scenes. Empirical results across counting, jigsaw solving, object placement, and multi-object hallucination show substantial performance gains over baselines, outperforming purely textual reasoning and visual prompting strategies. The findings highlight the importance of task decomposition in visual reasoning and suggest opportunities for broader application and efficiency improvements in open-world multimodal reasoning systems.

Abstract

Under pure textual modality, Large Language Models (LLMs) have demonstrated remarkable success in complex reasoning tasks by decomposing them into simpler sub-problems. However, Multimodal Large Language Models (MLLMs) still struggle with some seemingly straightforward visual tasks, such as counting and solving jigsaw puzzles. We argue that these tasks challenge the ability of visual-to-textual conversion, where MLLMs convert visual information perceived from the input scene, to textual information for further reasoning and generating the answer. If the complexity of the visual input is beyond the perceptual capability of the MLLMs, without decomposing this conversion process, simply scaling inference-time reasoning cannot solve the task because it repeatedly encounters the same perceptual bottleneck. We propose an approach, autonomous imagination, to enable MLLMs to iteratively modify visual inputs (e.g. isolating objects, rearranging puzzle pieces) into intermediate visual states, decomposing visual-to-textual conversion into closed-loop visual modification steps. We show that, without any retraining, MLLMs can now solve tasks initially beyond their perceptual capability, highlighting that closed-loop visual modification can be an effective way of decomposing the visual reasoning task into solvable substeps. Our code and data are released at https://future-item.github.io/autoimagine-site/.

Figures (16)

• Figure 1: Our autonomous imagination approach empowers advanced MLLMs to engage in iterative imaginative reasoning, enabling them to address previously unsolvable tasks without additional training.
• Figure 2: An overview of our autonomous imagination approach: The imagination space begins with an unstructured input scene and undergoes an iterative reasoning process. In each cycle, MLLMs first perceive the current state of the imagination space, select an operation to apply, and then reassess the updated imagination space. Upon completing this reasoning sequence, MLLMs generate an answer based on the cumulative context of the process and the final state of the imagination space.
• Figure 3: Illustrations of operations within our imagination space: transformations can be applied to focused elements (including the virtual cursor), focus operations allow segmentation of cursor-selected elements, and ignore operations make cursor-selected elements visually invisible.
• Figure 4: (a)(b) show that as counting task difficulty increases linearly, scaling inference-time textual reasoning (implemented as GPT-4o-text-cot, See Sec. \ref{['subsec:exp_counting']}) fails—and even performs worse than vanilla GPT-4o—as complexity exceeds perception limits. In contrast, our methods remain unaffected by these limits, achieving correct counting even as difficulty rises. Additionally, adding visual prompts can introduce noise, causing MLLMs to enter hallucination loops; for instance, in (c), the model incorrectly concluded there were 196 balls when only two were present. The qualitative results of counting are provided in the appendix.
• Figure 5: Qualitative comparison on simple jigsaw puzzle solving, we use a black background to make jigsaw pieces more visible to MLLMs. We illustrate the final visual state achieved by different methods after completing their reasoning processes and producing a solution. For clarity, the actual location of each coordinate in the image is highlighted with an orange line and circle.