Top-Down Compression: Revisit Efficient Vision Token Projection for Visual Instruction Tuning

TL;DR

This work addresses the inefficiency of vision-to-language projection in visual instruction tuning by introducing a Top-Down Compression framework. It combines Flash Global Fusion, which uses linear-time state-space models for global context propagation with a local-to-single scan, and Visual-Native Selection, which fuses a visual expert and an instruction-aware native expert to select a compact, informative token set. The approach yields 75%–95% reductions in visual tokens while achieving competitive or superior performance across 12 benchmarks, demonstrating significant practical gains in efficiency. The method enables high-resolution vision-language understanding with reduced memory and latency, facilitating deployment in real-world, resource-constrained settings.

Abstract

Visual instruction tuning aims to enable large language models to comprehend the visual world, with a pivotal challenge lying in establishing an effective vision-to-language projection. However, existing methods often grapple with the intractable trade-off between accuracy and efficiency. In this paper, we present LLaVA-Meteor, a novel approach designed to break this deadlock, equipped with a novel Top-Down Compression paradigm that strategically compresses visual tokens without compromising core information. Specifically, we construct a trainable Flash Global Fusion module based on efficient selective state space operators, which aligns the feature space while enabling each token to perceive holistic visual context and instruction preference at low cost. Furthermore, a local-to-single scanning manner is employed to effectively capture local dependencies, thereby enhancing the model's capability in vision modeling. To alleviate computational overhead, we explore a Visual-Native Selection mechanism that independently assesses token significance by both the visual and native experts, followed by aggregation to retain the most critical subset. Extensive experiments show that our approach reduces visual tokens by 75--95% while achieving comparable or superior performance across 12 benchmarks, significantly improving efficiency.

Figures (4)

• Figure 1: Illustration of visual token compression strategies. (a) Naive and learnable fusion methods reduce token count via predefined rules or learned aggregation, but may lose structural fidelity or require heavy training. (b) Vision-based selection filters important tokens using attention, yet ignores instruction relevance. (c) Our Top-Down Compression combines Flash Global Fusion for global context propagation and instruction preference summarization, with Visual-Native Selection to jointly evaluate token importance from visual and native perspectives.
• Figure 2: Overview of the proposed LLaVA-Meteor, which adopts a two-stage Top-Down Compression pipeline to efficiently project visual tokens for instruction-following tasks. An input image is first encoded by a vision encoder into dense visual tokens, including a class token (CLS). We omitted the partitioning step for a clearer presentation. The Flash Global Fusion module (Top) propagates semantic context and instruction-level priors through efficient global interactions, enhanced by a local-to-single scanning strategy that improves local awareness. A learnable instruction token (INS) is inserted to aggregate task-relevant cues during fusion. Subsequently, the Visual-Native Selection module (Down) evaluates token importance from two perspectives: a class-based visual expert and an instruction-based native expert. Their scores are aggregated to select a compact set of key tokens, which are passed to the large language model along with the input to generate the final response.
• Figure 3: Qualitative comparison of LLaVA-UHD and our LLaVA-Meteor.
• Figure 4: Qualitative comparisons for different expert selection strategies.