Latent Visual Reasoning

TL;DR

Latent Visual Reasoning (LVR) introduces a paradigm shift by enabling autoregressive reasoning directly in the visual embedding space of multimodal language models. The approach interleaves a latent visual reasoning phase, triggered by special tokens, with standard text generation and is trained via a two-stage process: supervised finetuning with a joint reconstruction/generation objective and reinforcement learning using a latent-adapted GRPO. Empirical results show state-of-the-art performance on perception-intensive visual question answering benchmarks, with notable gains on fine-grained visual understanding tasks and robust performance without relying on external visual-editing tools. The work is complemented by decoding strategies, ablations, and a reproducibility plan, underscoring LVR’s potential to unify visual and textual signals in multimodal reasoning.

Abstract

Multimodal Large Language Models (MLLMs) have achieved notable gains in various tasks by incorporating Chain-of-Thought (CoT) reasoning in language spaces. Recent work extends this direction by leveraging external tools for visual editing, thereby enhancing the visual signal along the reasoning trajectories. Nevertheless, these approaches remain fundamentally constrained: reasoning is still confined to the language space, with visual information treated as static preconditions. We introduce Latent Visual Reasoning (LVR), a new paradigm that enables autoregressive reasoning directly in the visual embedding space. A visual encoder first projects images into visual tokens within a joint semantic space shared with the language model. The language model is then trained to generate latent states that reconstruct key visual tokens critical for answering the query, constituting the process of latent visual reasoning. By interleaving LVR with standard text generation, our model achieves substantial gains on perception-intensive visual question answering tasks. In addition, we adapt the GRPO algorithm to conduct reinforcement learning on latent reasoning, further balancing LVR and textual generation. We show that LVR substantially improves fine-grained visual understanding and perception, achieving 71.67% on MMVP compared to 66.67% with Qwen2.5-VL. Code base and model weights will be released later.

Figures (2)

• Figure 1: Conceptual illustration of Latent Visual Reasoning (LVR). We compare LVR with two paradigms: "Think about images," which performs multimodal reasoning entirely in text space, and "Think with images", which leverages external visual tools to highlight regions of interest (ROIs). In contrast, LVR leverages the LLM’s latent space to reconstruct the semantics of ROIs, enabling seamless cross-modal reasoning.
• Figure 2: Training and inference pipeline of LVR. The overall framework closely follows a standard MLLM. Images are encoded into tokens by a visual encoder and mapped into a joint semantic space with text embeddings. During the SFT stage, bounding boxes are provided to identify query-relevant visual tokens, which supervise the last hidden states in the LVR process. Here, only the LLM’s last hidden states are passed forward for latent reasoning, optimized with a Mean Squared Error loss. The LVR process is wrapped with special tokens that indicate reasoning mode. Once all query-relevant visual tokens are consumed, the model exits LVR and resumes standard text generation with cross-entropy loss. During RL training, the model self-evolves the LVR process learned in SFT, while only the text generation part is supervised, using our adapted $GRPO_{latent}$. At inference, the model triggers LVR upon generating the special token, propagates hidden states to reconstruct visual semantics, and resumes text generation when a stopping criterion is met.