ViThinker: Active Vision-Language Reasoning via Dynamic Perceptual Querying

TL;DR

ViThinker tackles the Premature Visual-to-Text CoT limitation in vision-language models by enabling active perceptual querying. It internalizes vision experts through alignment, and employs a two-stage curriculum to learn when to look via task-driven query tokens, forming a Think-Query-Simulate-Think loop without external tools. A sparsity penalty enforces a minimal sufficient perceptual budget, and losses like $\mathcal{L}_{CE}$, $\mathcal{L}_{vis}$, and $\mathcal{L}_{p}$ guide learning across multiple valid reasoning paths. Empirical results across vision-centric benchmarks show ViThinker consistently outperforms passive baselines and other interleaved methods, with strong gains on fine-grained perception and high-resolution tasks, demonstrating the practical value of active, reasoning-driven perception in multimodal systems.

Abstract

Chain-of-Thought (CoT) reasoning excels in language models but struggles in vision-language models due to premature visual-to-text conversion that discards continuous information such as geometry and spatial layout. While recent methods enhance CoT through static enumeration or attention-based selection, they remain passive, i.e., processing pre-computed inputs rather than actively seeking task-relevant details. Inspired by human active perception, we introduce ViThinker, a framework that enables vision-language models to autonomously generate decision (query) tokens triggering the synthesis of expert-aligned visual features on demand. ViThinker internalizes vision-expert capabilities during training, performing generative mental simulation during inference without external tool calls. Through a two-stage curriculum: first distilling frozen experts into model parameters, then learning task-driven querying via sparsity penalties, i.e., ViThinker discovers minimal sufficient perception for each reasoning step. Evaluations across vision-centric benchmarks demonstrate consistent improvements, validating that active query generation outperforms passive approaches in both perceptual grounding and reasoning accuracy.

Figures (6)

• Figure 1: Overview of ViThinker Framework. ViThinker enables interleaved vision-language reasoning through a "Think-Query-Simulate-Think" loop. Left: Given a Visual-QA input, the VLM processes image and text into tokens. Middle: During training, we distill frozen vision experts (SAM, DepthAnything, PIDINet, DINOv2) into the VLM via multi-task supervision. Right: At inference, ViThinker performs internalized simulation. Crucially, no external vision models are invoked; instead, the generated query tokens (e.g., <query_seg>) directly trigger the synthesis of expert-aligned features from the model's parameters.
• Figure 2: Stage 1 focuses on skill acquisition. The model learns to encode and synthesize visual features by observing expert outputs provided in the context.
• Figure 3: Stage 2 optimizes the decision policy. By presenting multiple valid reasoning paths with different perceptual coverage, the sparsity penalty guides the model toward task-appropriate feature selection.
• Figure 4: Qualitative comparison of reasoning paradigms. Text CoT (left) lacks perceptual grounding. Sequential CoVT (middle) passively generates statistically frequent patterns (seg+patch) learned from training, while ViThinker (right) actively selects task-driven tokens (depth+seg) based on spatial reasoning requirements.
• Figure 5: Effect of tokens per expert ($N$) on CV-Bench performance (left Y-axis) and inference time cost (right Y-axis).