Imagination Helps Visual Reasoning, But Not Yet in Latent Space

TL;DR

This work investigates the validity of latent reasoning using Causal Mediation Analysis and proposes a straightforward alternative named CapImagine, which teaches the model to explicitly imagine using text, highlighting the superior potential of visual reasoning through explicit imagination.

Abstract

Latent visual reasoning aims to mimic human's imagination process by meditating through hidden states of Multimodal Large Language Models. While recognized as a promising paradigm for visual reasoning, the underlying mechanisms driving its effectiveness remain unclear. Motivated to demystify the true source of its efficacy, we investigate the validity of latent reasoning using Causal Mediation Analysis. We model the process as a causal chain: the input as the treatment, the latent tokens as the mediator, and the final answer as the outcome. Our findings uncover two critical disconnections: (a) Input-Latent Disconnect: dramatic perturbations on the input result in negligible changes to the latent tokens, suggesting that latent tokens do not effectively attend to the input sequence. (b) Latent-Answer Disconnect: perturbations on the latent tokens yield minimal impact on the final answer, indicating the limited causal effect latent tokens imposing on the outcome. Furthermore, extensive probing analysis reveals that latent tokens encode limited visual information and exhibit high similarity. Consequently, we challenge the necessity of latent reasoning and propose a straightforward alternative named CapImagine, which teaches the model to explicitly imagine using text. Experiments on vision-centric benchmarks show that CapImagine significantly outperforms complex latent-space baselines, highlighting the superior potential of visual reasoning through explicit imagination.

Figures (6)

• Figure 1: Comparison between visual reasoning with tools and through imagination. (a) Reasoing with tools perceive visual content through function calling such as zoom-in or drawing. (b) Latent-space imagination exploits the hidden states of MLLMs to conduct visual reasoning. (c) We show that imagination can be more effective in text-space.
• Figure 2: Our systematic latent analysis framework for investigating the internal mechanisms and behavioral patterns of latent tokens. (a) Model Inference illustrates the latent inference process. (b) and (c) respectively illustrate two causal analysis approaches. In diagram Intervention on $Z$, $\tau$ denotes a fixed tensor, $\epsilon$ represents random Gaussian noise with $\epsilon \sim \mathcal{N}(0, \sigma^2)$, and $\mu$ is a small value close to zero.
• Figure 3: Illustration of Our Method and Data Construction Pipeline, through which we conduct a strictly controlled training setting with Monet for fair and convincing comparisons. The upper section presents the interleaved format of original data. The middle section clarifies the key methodological differences between the two approaches. The lower section shows the data construction procedures.
• Figure 4: Inter-instance and Intra-instance Analysis of the inner hidden states of CapImagine during reasoning process.
• Figure 5: Inference Speed Comparison of Monet, CapImagine and DeepEyes on V*. (Unit: Seconds)