VCM: Vision Concept Modeling Based on Implicit Contrastive Learning with Vision-Language Instruction Fine-Tuning

TL;DR

This work addresses the inefficiency of LVLMs that process full images token-by-token by introducing Vision Concept Modeling (VCM), a self-supervised framework that dynamically extracts and aligns vision concepts to textual instructions. Through implicit contrastive sampling, a keyword-based semantic alignment, and a forward-backward dynamic programming module, VCM learns concept-level representations without expensive annotations and tunes the vision output length via a differentiable DP mechanism. Empirically, VCM achieves up to $85\%$ FLOPs reduction while maintaining strong performance on VQA, and it enhances open-vocabulary detection and segmentation, as well as dense perception in the vision encoder, across multiple benchmarks and modalities. The method demonstrates broad applicability, scalability across architectures, and potential for deployment on resource-constrained platforms, though it relies on adaptive keyword selection that may introduce biases and benefits from further refinement in keyword extraction and length estimation.

Abstract

Large Vision-Language Models (LVLMs) are pivotal for real-world AI tasks like embodied intelligence due to their strong vision-language reasoning abilities. However, current LVLMs process entire images at the token level, which is inefficient compared to humans who analyze information and generate content at the conceptual level, extracting relevant visual concepts with minimal effort. This inefficiency, stemming from the lack of a visual concept model, limits LVLMs' usability in real-world applications. To address this, we propose VCM, an end-to-end self-supervised visual concept modeling framework. VCM leverages implicit contrastive learning across multiple sampled instances and vision-language fine-tuning to construct a visual concept model without requiring costly concept-level annotations. Our results show that VCM significantly reduces computational costs (e.g., 85\% fewer FLOPs for LLaVA-1.5-7B) while maintaining strong performance across diverse image understanding tasks. Moreover, VCM enhances visual encoders' capabilities in classic visual concept perception tasks. Extensive quantitative and qualitative experiments validate the effectiveness and efficiency of VCM.

Figures (10)

• Figure 1: Illustrations of VCM in enhancing efficiency and dense perception capability. VCM can select relevant vision concept tokens based on instructions, significantly reducing redundant attention computations in LVLMs. This reduction lowers both training and inference costs while maintaining strong performance, as shown in (a). Additionally, VCM enhances the dense concept prediction capability of the vision encoder, as illustrated in (b) through K-Means visualizations of dense feature maps from the last layer of CLIP ViT. These improvements enable broader applicability to dense perception vision-language tasks.
• Figure 2: Correlations between vision tokens and text prior. (a) Positive correlation between the text response and minimum vision tokens: the more image-related keywords in the text response, the longer the minimum token length required. (b) Negative correlation between the text instruction and minimum vision tokens: the more image-related keywords in the text instruction, the shorter the minimum token length required. (c) Negative correlation between the difference in text response and instruction and minimum vision tokens: the greater the gap in image-related keywords between the text response and instruction, the shorter the minimum vision token length required.
• Figure 3: Overview of our VCM framework. (a) The workflow architecture: vision concepts are extracted from image inputs based on instruction priors and fed into the LLM to generate corresponding answers. (b) Adaptive keyword selection module: image-relevant keywords (highlighted in red) are selected by calculating text-image similarity, retaining keywords with scores above the average. (c) Implicit contrastive sampling module: keywords in the instruction are randomly masked, and the VCM loss is computed with input image features for end-to-end optimization.
• Figure 4: Visualization results about our VCM. (Top) Visualization of VCM with different instructions. From left to right, the visual representation becomes increasingly sparse, leaving corresponding vision tokens to unmasked keywords (highlighted in red). (Bottom) K-Means visualization of dense feature maps of CLIP ViT. We show the raw images, the K-Means results without VCM, and those of our fine-tuned model by VCM.
• Figure 5: Comparison of three different vision token reduction paradigms. (a) Regular training-free token pruning method based on threshold filtering of the attention matrix. (b) Regular token merging method based on the attention mechanism and fixed-length trainable query vectors. (c) Our method, through adaptive local merging and filtering modeled by VCM, allows precise control of the vision concept length while keeping the positional information.

Theorems & Definitions (1)

• proof