Latent Implicit Visual Reasoning

TL;DR

This work addresses the gap where Large Multimodal Models remain overly text-centric for vision-centric reasoning and rely on costly, task-specific supervision. It introduces Latent Implicit Visual Reasoning (LIVR), which augments LMMs with $K$ latent tokens and a visual bottleneck that forces visual information to flow through these latents, learned via a two-stage training regime without explicit intermediate supervision. LIVR achieves state-of-the-art performance on a diverse set of perception-heavy tasks and generalizes to multi-task fine-tuning, outperforming direct supervised fine-tuning across multiple backbones and tasks. The approach demonstrates that task-agnostic, latent visual representations can be learned end-to-end, providing a scalable path to richer visual reasoning in multimodal models.

Abstract

While Large Multimodal Models (LMMs) have made significant progress, they remain largely text-centric, relying on language as their core reasoning modality. As a result, they are limited in their ability to handle reasoning tasks that are predominantly visual. Recent approaches have sought to address this by supervising intermediate visual steps with helper images, depth maps, or image crops. However, these strategies impose restrictive priors on what "useful" visual abstractions look like, add heavy annotation costs, and struggle to generalize across tasks. To address this critical limitation, we propose a task-agnostic mechanism that trains LMMs to discover and use visual reasoning tokens without explicit supervision. These tokens attend globally and re-encode the image in a task-adaptive way, enabling the model to extract relevant visual information without hand-crafted supervision. Our approach outperforms direct fine-tuning and achieves state-of-the-art results on a diverse range of vision-centric tasks -- including those where intermediate abstractions are hard to specify -- while also generalizing to multi-task instruction tuning.

Figures (5)

• Figure 1: The model is asked to determine which image option is most similar to the reference image. Standard LMMs can only output text, which cannot capture all visual information and may introduce ambiguity. While methods using explicit supervision can train models to output intermediate reasoning steps, these approaches may fail when the reasoning steps themselves are unclear. Our approach allows the model to learn useful representations implicitly. Visualizing the attention maps of the latent tokens shows that the model has learned to recognize underlying visual structures relevant to answering the question that would have been hard for humans to design supervision for.
• Figure 2: An illustration of our method and bottleneck attention masking. Latent tokens are appended to the prompt and losses are computed on the answer tokens. In our bottleneck attention masking, answers and prompt tokens cannot attend to image tokens.
• Figure 3: An illustration of latent-to-image attention maps for different tasks. The left columns show the input images, and the right columns show the attention overlays. In the Semantic Correspondence task, the model identifies the option in the second image that aligns with the REF point in the first image. In the Localization task, it selects bounding boxes that best localize the motorcycle and the dog, and in the Counting task, it counts the cows and balloons. We observe that latent-to-image attention concentrates on regions corresponding to the correct answers or the visual evidence needed to resolve each task. Although some attention sinks persist, the dominant patterns align with task-relevant regions, indicating that the latents capture meaningful visual structure without explicit supervision.
• Figure 4: t-SNE Visualization of Different Tokens
• Figure 5: Additional visualizations of latent token to image attention.