Reasoning in the Dark: Interleaved Vision-Text Reasoning in Latent Space

TL;DR

The paper tackles the inefficiency and annotation burden of explicit multimodal reasoning in multimodal LLMs by introducing Interleaved Vision-Text Latent Reasoning (IVT-LR), which conducts reasoning entirely in a multimodal latent space using latent text and latent vision. It combines a progressive multi-stage training strategy that gradually replaces explicit chain-of-thought steps with latent steps, significantly reducing autoregressive steps and inference latency while maintaining or improving accuracy on M$^3$CoT and ScienceQA. The approach achieves state-of-the-art results and demonstrates the importance of jointly modeling latent textual and visual information, validated through extensive ablations and analyses of latent text, latent vision, and attention dynamics. This latent-space paradigm offers a data-efficient, scalable path toward faster and more capable vision-language reasoning in VQA and related tasks.

Abstract

Multimodal reasoning aims to enhance the capabilities of MLLMs by incorporating intermediate reasoning steps before reaching the final answer. It has evolved from text-only reasoning to the integration of visual information, enabling the thought process to be conveyed through both images and text. Despite its effectiveness, current multimodal reasoning methods depend on explicit reasoning steps that require labor-intensive vision-text annotations and inherently introduce significant inference latency. To address these issues, we introduce multimodal latent reasoning with the advantages of multimodal representation, reduced annotation, and inference efficiency. To facilicate it, we propose Interleaved Vision-Text Latent Reasoning (IVT-LR), which injects both visual and textual information in the reasoning process within the latent space. Specifically, IVT-LR represents each reasoning step by combining two implicit parts: latent text (the hidden states from the previous step) and latent vision (a set of selected image embeddings). We further introduce a progressive multi-stage training strategy to enable MLLMs to perform the above multimodal latent reasoning steps. Experiments on M3CoT and ScienceQA demonstrate that our IVT-LR method achieves an average performance increase of 5.45% in accuracy, while simultaneously achieving a speed increase of over 5 times compared to existing approaches. Code available at https://github.com/FYYDCC/IVT-LR.

Figures (6)

• Figure 1: An example of interleaved vision-text latent reasoning, where the intermediate reasoning steps are carried out entirely within the multimodal latent space.
• Figure 2: Overview of our Interleaved Vision-Text Latent Reasoning (IVT-LR) framework. At each step, reasoning is performed entirely in the latent space by fusing $\emph{latent text}$ (the hidden state from the previous step) and $\emph{latent vision}$ (dynamically selected image embeddings based on attention scores).
• Figure 3: Overview of the Multi-Stage Progressive Training Strategy used for IVT-LR. The strategy begins with full explicit CoT and then gradually substitutes one explicit reasoning step with latent text and latent vision. Training loss is calculated exclusively over the remaining explicit steps and the final answer.
• Figure 4: Accuracy comparison of IVT-LR on the length of latent vision per reasoning step across two reasoning benchmarks: (a) M$^3$CoT and (b) ScienceQA.
• Figure 5: Attention analysis comparison between explicit and latent reasoning approaches. Left: attention ratios of visual part to textual part across reasoning steps. Right: attention focus measured by inverse entropy.