Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LaViT: Aligning Latent Visual Thoughts for Multi-modal Reasoning

Linquan Wu, Tianxiang Jiang, Yifei Dong, Haoyu Yang, Fengji Zhang, Shichaang Meng, Ai Xuan, Linqi Song, Jacky Keung

TL;DR

This work tackles the Perception Gap in multimodal distillation, where student models imitate teacher outputs without grounding their reasoning in the visual evidence. It introduces LaViT, a framework that distills latent visual thoughts through white-box trajectory distillation and a curriculum sensory gating mechanism, pushing the student to reconstruct the teacher’s visual semantics and attention trajectories before generating textual responses. Empirical results show substantial gains in visual grounding and complex reasoning, with LaViT-3B outperforming larger open-source models and rivaling proprietary systems on several benchmarks. The approach offers a parameter-efficient path to robust multimodal reasoning by ensuring alignment between what is seen and how it is attended during reasoning.

Abstract

Current multimodal latent reasoning often relies on external supervision (e.g., auxiliary images), ignoring intrinsic visual attention dynamics. In this work, we identify a critical Perception Gap in distillation: student models frequently mimic a teacher's textual output while attending to fundamentally divergent visual regions, effectively relying on language priors rather than grounded perception. To bridge this, we propose LaViT, a framework that aligns latent visual thoughts rather than static embeddings. LaViT compels the student to autoregressively reconstruct the teacher's visual semantics and attention trajectories prior to text generation, employing a curriculum sensory gating mechanism to prevent shortcut learning. Extensive experiments show that LaViT significantly enhances visual grounding, achieving up to +16.9% gains on complex reasoning tasks and enabling a compact 3B model to outperform larger open-source variants and proprietary models like GPT-4o.

LaViT: Aligning Latent Visual Thoughts for Multi-modal Reasoning

TL;DR

This work tackles the Perception Gap in multimodal distillation, where student models imitate teacher outputs without grounding their reasoning in the visual evidence. It introduces LaViT, a framework that distills latent visual thoughts through white-box trajectory distillation and a curriculum sensory gating mechanism, pushing the student to reconstruct the teacher’s visual semantics and attention trajectories before generating textual responses. Empirical results show substantial gains in visual grounding and complex reasoning, with LaViT-3B outperforming larger open-source models and rivaling proprietary systems on several benchmarks. The approach offers a parameter-efficient path to robust multimodal reasoning by ensuring alignment between what is seen and how it is attended during reasoning.

Abstract

Current multimodal latent reasoning often relies on external supervision (e.g., auxiliary images), ignoring intrinsic visual attention dynamics. In this work, we identify a critical Perception Gap in distillation: student models frequently mimic a teacher's textual output while attending to fundamentally divergent visual regions, effectively relying on language priors rather than grounded perception. To bridge this, we propose LaViT, a framework that aligns latent visual thoughts rather than static embeddings. LaViT compels the student to autoregressively reconstruct the teacher's visual semantics and attention trajectories prior to text generation, employing a curriculum sensory gating mechanism to prevent shortcut learning. Extensive experiments show that LaViT significantly enhances visual grounding, achieving up to +16.9% gains on complex reasoning tasks and enabling a compact 3B model to outperform larger open-source variants and proprietary models like GPT-4o.
Paper Structure (39 sections, 10 equations, 5 figures, 7 tables)

This paper contains 39 sections, 10 equations, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Visual Chain-of-Thought
  4. Latent Reasoning
  5. Knowledge Distillation
  6. Empirical Analysis of Perception Gap
  7. Visual Attention Dictates Reasoning Bounds
  8. Perception Gap between Teachers and Students
  9. Analysis of Attention Divergence.
  10. Method
  11. LaViT-SFT-15K
  12. White-box Trajectory Distillation
  13. Curriculum Sensory Gating
  14. Optimization Objectives
  15. Joint Training Dynamics
  16. ...and 24 more sections

Figures (5)

  • Figure 1: Conceptual Illustration of Our Proposed Method LaViT.
  • Figure 2: Impact of Visual Attention on Reasoning Accuracy. The monotonic increase in accuracy with higher Visual Focusing Score ($S_{focus}$) thresholds validates that effective visual grounding is a prerequisite for correct reasoning.
  • Figure 3: The Perception-Reasoning Gap. While the student aligns closely with the teacher in textual representations (stable Cosine Distance), their visual attention trajectories diverge significantly on attribute-heavy tokens (rising KL Divergence). This reveals that textual mimicry does not imply visual grounding.
  • Figure 4: Attention entropy distribution.
  • Figure 5: Visualization of attention distributions across Qwen2.5-VL-3B, 32B (Teacher), and LaViT on two representative samples from the BLINK. $\blacktriangle$ indicates the task-relevant critical regions required for correct reasoning.