Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Modal-Mixed Chain-of-Thought Reasoning with Latent Embeddings

Yifei Shao, Kun Zhou, Ziming Xu, Mohammad Atif Quamar, Shibo Hao, Zhen Wang, Zhiting Hu, Biwei Huang

TL;DR

This work introduces modal-mixed chain-of-thought (CoT) for vision-language models, enabling reasoning to interleave text with compact latent visual sketches. A diffusion-based latent decoder, conditioned on the VLM's hidden states, carries fine-grained perceptual details while the language backbone provides high-level intent, with a diffusion-aligned latent space established by training the VLM to reconstruct its own visual embeddings. The model is trained in two stages—supervised fine-tuning on interleaved text-and-latent traces and reinforcement learning to optimize modality switching—leading to consistent gains on 11 multimodal tasks across perception and reasoning dimensions. The approach reduces dependency on external tools, improves grounding, and scales to longer reasoning chains, demonstrating practical advantages for multimodal reasoning. The findings suggest a promising direction for integrating latent visual reasoning with language models in a way that preserves core linguistic capabilities while expanding multimodal grounding.

Abstract

We study how to extend chain-of-thought (CoT) beyond language to better handle multimodal reasoning. While CoT helps LLMs and VLMs articulate intermediate steps, its text-only form often fails on vision-intensive problems where key intermediate states are inherently visual. We introduce modal-mixed CoT, which interleaves textual tokens with compact visual sketches represented as latent embeddings. To bridge the modality gap without eroding the original knowledge and capability of the VLM, we use the VLM itself as an encoder and train the language backbone to reconstruct its own intermediate vision embeddings, to guarantee the semantic alignment of the visual latent space. We further attach a diffusion-based latent decoder, invoked by a special control token and conditioned on hidden states from the VLM. In this way, the diffusion head carries fine-grained perceptual details while the VLM specifies high-level intent, which cleanly disentangles roles and reduces the optimization pressure of the VLM. Training proceeds in two stages: supervised fine-tuning on traces that interleave text and latents with a joint next-token and latent-reconstruction objective, followed by reinforcement learning that teaches when to switch modalities and how to compose long reasoning chains. Extensive experiments across 11 diverse multimodal reasoning tasks, demonstrate that our method yields better performance than language-only and other CoT methods. Our code will be publicly released.

Learning Modal-Mixed Chain-of-Thought Reasoning with Latent Embeddings

TL;DR

This work introduces modal-mixed chain-of-thought (CoT) for vision-language models, enabling reasoning to interleave text with compact latent visual sketches. A diffusion-based latent decoder, conditioned on the VLM's hidden states, carries fine-grained perceptual details while the language backbone provides high-level intent, with a diffusion-aligned latent space established by training the VLM to reconstruct its own visual embeddings. The model is trained in two stages—supervised fine-tuning on interleaved text-and-latent traces and reinforcement learning to optimize modality switching—leading to consistent gains on 11 multimodal tasks across perception and reasoning dimensions. The approach reduces dependency on external tools, improves grounding, and scales to longer reasoning chains, demonstrating practical advantages for multimodal reasoning. The findings suggest a promising direction for integrating latent visual reasoning with language models in a way that preserves core linguistic capabilities while expanding multimodal grounding.

Abstract

We study how to extend chain-of-thought (CoT) beyond language to better handle multimodal reasoning. While CoT helps LLMs and VLMs articulate intermediate steps, its text-only form often fails on vision-intensive problems where key intermediate states are inherently visual. We introduce modal-mixed CoT, which interleaves textual tokens with compact visual sketches represented as latent embeddings. To bridge the modality gap without eroding the original knowledge and capability of the VLM, we use the VLM itself as an encoder and train the language backbone to reconstruct its own intermediate vision embeddings, to guarantee the semantic alignment of the visual latent space. We further attach a diffusion-based latent decoder, invoked by a special control token and conditioned on hidden states from the VLM. In this way, the diffusion head carries fine-grained perceptual details while the VLM specifies high-level intent, which cleanly disentangles roles and reduces the optimization pressure of the VLM. Training proceeds in two stages: supervised fine-tuning on traces that interleave text and latents with a joint next-token and latent-reconstruction objective, followed by reinforcement learning that teaches when to switch modalities and how to compose long reasoning chains. Extensive experiments across 11 diverse multimodal reasoning tasks, demonstrate that our method yields better performance than language-only and other CoT methods. Our code will be publicly released.
Paper Structure (31 sections, 5 equations, 5 figures, 8 tables)

This paper contains 31 sections, 5 equations, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision-language Models.
  4. Chain-of-thought Reasoning.
  5. Latent Reasoning.
  6. Preliminary
  7. Vision Language Models.
  8. Chain-of-thought.
  9. Approach
  10. Model Architecture
  11. Diffusion Model based Decoder.
  12. Modal-mixed Chain-of-thought Reasoning.
  13. Learning to Reason with Latent Embeddings
  14. Supervised Fine-tuning with VLM as Visual Latent Encoder.
  15. Reinforcement Learning for Self-adaptation.
  16. ...and 16 more sections

Figures (5)

  • Figure 1: The overview of our proposed method. We integrate the diffusion model based decoder into the VLM and train it to learn the interleaved modal-mixed CoT reasoning paradigm.
  • Figure 2: Visualization of Visual Tokens Attention. The heatmaps (Layer 14) are shown above their respective QA pairs. The model successfully focuses on the semantic regions relevant to the question.
  • Figure 3: Case Study 1: Fine-grained Attribute Recognition. Our model generates visual tokens to focus on the person, correctly mapping the visual attribute to Option B. The baseline model suffers from hallucination between the visual perception and the option selection.
  • Figure 4: Case Study 2: Abstract Logic Reasoning. Our model explicitly visualizes the state of symbols after each transformation step using latent visual tokens, enabling it to track the multi-step logic correctly.
  • Figure 5: Case Study 3: Spatial Fitting. By mentally "placing" the piece into the options via latent visualization, our model correctly identifies the geometric correspondence, whereas the baseline relies on erroneous textual descriptions of the shapes.