Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens

TL;DR

The paper addresses the gap in vision-language models' perceptual understanding by introducing Chain-of-Visual-Thought (CoVT), which injects continuous visual tokens representing segmentation, depth, edges, and DINO features into the model's reasoning process. CoVT trains the model to predict these tokens through a four-stage pipeline and aligns them with lightweight vision experts, enabling reasoning directly in a dense visual latent space while optionally decoding to dense predictions for interpretability. Empirical results show consistent, substantial gains across over ten perception benchmarks, with notable improvements in depth perception and fine-grained visual reasoning, and qualitative analyses demonstrate meaningful, interpretable visual thought components. The work highlights a path toward more grounded, efficient, and interpretable multimodal intelligence by combining latent visual reasoning with conventional language-based VLM capabilities.

Abstract

Vision-Language Models (VLMs) excel at reasoning in linguistic space but struggle with perceptual understanding that requires dense visual perception, e.g., spatial reasoning and geometric awareness. This limitation stems from the fact that current VLMs have limited mechanisms to capture dense visual information across spatial dimensions. We introduce Chain-of-Visual-Thought (COVT), a framework that enables VLMs to reason not only in words but also through continuous visual tokens-compact latent representations that encode rich perceptual cues. Within a small budget of roughly 20 tokens, COVT distills knowledge from lightweight vision experts, capturing complementary properties such as 2D appearance, 3D geometry, spatial layout, and edge structure. During training, the VLM with COVT autoregressively predicts these visual tokens to reconstruct dense supervision signals (e.g., depth, segmentation, edges, and DINO features). At inference, the model reasons directly in the continuous visual token space, preserving efficiency while optionally decoding dense predictions for interpretability. Evaluated across more than ten diverse perception benchmarks, including CV-Bench, MMVP, RealWorldQA, MMStar, WorldMedQA, and HRBench, integrating COVT into strong VLMs such as Qwen2.5-VL and LLaVA consistently improves performance by 3% to 16% and demonstrates that compact continuous visual thinking enables more precise, grounded, and interpretable multimodal intelligence.

Figures (17)

• Figure 1: Rather than restricting VLM reasoning to the discrete language space with limited representational capacity, CoVT forms a visual thought chain that enables VLMs to reason in continuous visual space. By introducing continuous visual tokens that encode perceptual cues (e.g., segmentation, depth, instance, and edge structure), CoVT composes chains of textual and visual thoughts that link semantic reasoning with perceptual grounding. These visual "thought chains" bridge language and vision, enabling fine-grained understanding, spatial precision, and geometric awareness beyond the reach of text-based reasoning.
• Figure 2: Continuous visual thinking with CoVT.CoVT introduces compact, continuous visual tokens that encode fine-grained perceptual cues, such as object localization, spatial structure, and scene semantics, directly into VLM reasoning. These tokens ground multimodal reasoning in visual space, enabling the model to capture fine-grained relationships across vision-centric tasks (e.g., counting, depth ordering, and scene understanding) without relying on external tools. They can also be decoded into dense predictions, offering human-interpretable visualizations of the model's reasoning process.
• Figure 3: The training pipeline of CoVT.CoVT first generates the thinking process, containing visual thinking tokens, and then leverages these visual thoughts to condition next token prediction and reason the final answer. To endow these tokens with perceptual meaning, we align them with lightweight vision experts (e.g., SAM, DepthAnything, PIDINet, DINO) on their respective tasks during training. Specifically: SAM uses 8 visual tokens as mask prompts; DepthAnything uses 4 tokens to reconstruct depth; PIDINet uses 4 tokens to reconstruct edges; and DINO uses 4 tokens to match patch-level features. The VLM is finetuned with LoRA and all the projection layers are trainable. Note: During inference, dense predictions are decoded only when interpretability is desired; otherwise, reasoning occurs entirely in the latent visual space.
• Figure 4: Four-stage data formatting for CoVT. The first stage helps the model comprehend the visual tokens, and the second stage guides it to generate them. The third stage enables the VLM to integrate visual tokens into its reasoning process, while the final stage allows the model to efficiently select and utilize visual thinking tokens within visual thought chains.
• Figure 5: Visualization of CoVT tokens. Different visual tokens contribute complementary cues that enable the model to solve complex perceptual reasoning tasks. Left: Segmentation tokens localize point B on the face, while the depth tokens capture the relative depth relationships. Mid: Depth visual tokens provide depth map information, and the edge tokens help highlight the positions of two boxes. Right: The Segmentation tokens identify the attended region, and the edge tokens delineate the fine-grained line structures.