Chain-of-Thought Compression Should Not Be Blind: V-Skip for Efficient Multimodal Reasoning via Dual-Path Anchoring

TL;DR

This work tackles the latency bottleneck of multimodal chain-of-thought by identifying Visual Amnesia, where text-centric pruning discards visually grounded tokens. It proposes Visual-Anchored Information Bottleneck (VA-IB) and a dual-path Visual Anchoring Score to guide V-Skip, which conserves visual anchors via a Union-of-Saliency gate and is distilled into a LoRA-adapted student for efficient inference. Empirical evaluation on Qwen2-VL and Llama-3.2 shows up to a 2.9× speedup with negligible accuracy loss, with pronounced gains on DocVQA and robust performance across model scales. The approach reduces hallucination risk by preserving visual grounding and offers a practical, grounding-aware compression framework for multimodal reasoning.

Abstract

While Chain-of-Thought (CoT) reasoning significantly enhances the performance of Multimodal Large Language Models (MLLMs), its autoregressive nature incurs prohibitive latency constraints. Current efforts to mitigate this via token compression often fail by blindly applying text-centric metrics to multimodal contexts. We identify a critical failure mode termed Visual Amnesia, where linguistically redundant tokens are erroneously pruned, leading to hallucinations. To address this, we introduce V-Skip that reformulates token pruning as a Visual-Anchored Information Bottleneck (VA-IB) optimization problem. V-Skip employs a dual-path gating mechanism that weighs token importance through both linguistic surprisal and cross-modal attention flow, effectively rescuing visually salient anchors. Extensive experiments on Qwen2-VL and Llama-3.2 families demonstrate that V-Skip achieves a $2.9\times$ speedup with negligible accuracy loss. Specifically, it preserves fine-grained visual details, outperforming other baselines over 30\% on the DocVQA.

Figures (6)

• Figure 1: Comparison of token compression paradigms. (Top) The original MLLMs retains all tokens, ensuring accuracy but incurring high computational costs. (Middle) Standard text-centric compression aggressively prunes the adjective red due to its high linguistic probability given apple, detaching the reasoning from the visual ground truth and leading to hallucination. (Bottom) V-Skip utilizes a dual-path scoring mechanism to identify and rescue visually salient tokens while compressing redundant linguistic fillers.
• Figure 2: The V-Skip Training Pipeline. Our method automates the construction of efficient multimodal reasoners through three distinct stages. (1) Data Generation, a VLM generates comprehensive CoT rationales for input image-question pairs. (2) Filtering & Pruning, Step A filters out rationales that lead to incorrect answers compared to the ground truth. Step B applies our Visual-Anchored Pruning mechanism (detailed in Figure \ref{['figure3']}), which selectively removes redundant tokens based on a target compression rate $\gamma$, producing concise Compressed_CoT sequences. (3) Efficient Fine-tuning, the resulting high-quality compressed dataset is used to supervise a Base Model via LoRA, yielding an Efficient Reasoner capable of generating succinct and visually grounded reasoning paths directly.
• Figure 3: The Visual-Anchored Token Pruning Mechanism. The pruning process evaluates each generated token through a dual-pathway scoring framework. The Textual Pathway quantifies the linguistic necessity of a token by identifying redundant functional fillers, such as "it" or "is" based on their language prior scores. The Visual Pathway computes the VAS by aggregating cross-modal attention weights, which quantifies the grounding strength of a token relative to specific image regions. Finally, the V-Skip Gate applies a union-of-saliency operation to determine token retention. A token is pruned only if it is diagnosed as both linguistically redundant and visually irrelevant.
• Figure 4: Performance robustness of V-Skip on Llama-3.2-11B-Vision-Instruct across varying compression ratios. The dual-axis plot displays the impact of the target compression ratio $\gamma$ (x-axis) on model performance. The left axis (blue circle) reports the Accuracy (%) for the MMMU, while the right axis (green square) indicates the ANLS Score for the DocVQA.
• Figure 5: Qualitative comparison on DocVQA illustrating the Information Entropy Mismatch. V-Skip successfully retains the high-surprisal token $45.20 by leveraging visual attention, whereas text-only pruning fails.