Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VT-CLIP: Enhancing Vision-Language Models with Visual-guided Texts

Longtian Qiu, Renrui Zhang, Ziyu Guo, Ziyao Zeng, Zilu Guo, Yafeng Li, Guangnan Zhang

TL;DR

This work tackles sub-optimal cross-modal alignment in CLIP caused by semantic gaps between image regions and text prompts. It introduces VT-CLIP, which refines text features through a visual-guided cross-attention module using intermediate image features, with backbones frozen. Evaluated on 11 datasets under few-shot regimes, VT-CLIP consistently outperforms Zero-shot CLIP and prompt-based baselines, while maintaining stability across data scales. The results indicate that instance-level visual context can tailor text representations to downstream images, strengthening cross-modal alignment and generalization.

Abstract

Contrastive Language-Image Pre-training (CLIP) has drawn increasing attention recently for its transferable visual representation learning. However, due to the semantic gap within datasets, CLIP's pre-trained image-text alignment becomes sub-optimal on downstream tasks, which severely harms its transferring performance. To better adapt the cross-modality embedding space, we propose to enhance CLIP via Visual-guided Texts, named VT-CLIP. Specifically, we guide textual features of different categories to adaptively explore informative regions on the image and aggregate visual features by attention mechanisms. In this way, the texts become visual-guided, namely, more semantically correlated with downstream images, which greatly benefits the category-wise matching process. In few-shot settings, we evaluate our VT-CLIP on 11 well-known classification datasets to demonstrate its effectiveness.

VT-CLIP: Enhancing Vision-Language Models with Visual-guided Texts

TL;DR

This work tackles sub-optimal cross-modal alignment in CLIP caused by semantic gaps between image regions and text prompts. It introduces VT-CLIP, which refines text features through a visual-guided cross-attention module using intermediate image features, with backbones frozen. Evaluated on 11 datasets under few-shot regimes, VT-CLIP consistently outperforms Zero-shot CLIP and prompt-based baselines, while maintaining stability across data scales. The results indicate that instance-level visual context can tailor text representations to downstream images, strengthening cross-modal alignment and generalization.

Abstract

Contrastive Language-Image Pre-training (CLIP) has drawn increasing attention recently for its transferable visual representation learning. However, due to the semantic gap within datasets, CLIP's pre-trained image-text alignment becomes sub-optimal on downstream tasks, which severely harms its transferring performance. To better adapt the cross-modality embedding space, we propose to enhance CLIP via Visual-guided Texts, named VT-CLIP. Specifically, we guide textual features of different categories to adaptively explore informative regions on the image and aggregate visual features by attention mechanisms. In this way, the texts become visual-guided, namely, more semantically correlated with downstream images, which greatly benefits the category-wise matching process. In few-shot settings, we evaluate our VT-CLIP on 11 well-known classification datasets to demonstrate its effectiveness.
Paper Structure (11 sections, 4 equations, 3 figures, 2 tables)

This paper contains 11 sections, 4 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. RELATED WORK
  3. METHODS
  4. Zero-shot CLIP
  5. VT-CLIP
  6. EXPERIMENTS
  7. Training Settings
  8. Performance Comparison & Analysis
  9. Performance Comparison & Analysis
  10. Ablation Study
  11. CONCLUSION

Figures (3)

  • Figure 1: Visualization of attention map and prediction score. In our VT-CLIP, the informative regions on the image gain more attention weights from the texts, which improves the prediction score on ground-truth category.
  • Figure 2: Structures of VT-CLIP. Class-level text feature from CLIP text encoder is updated by spatial visual image feature with cross attention module
  • Figure 3: Experiment Results––Main results of few-shot learning on 11 datasets. VT-CLIP shows overall better performance over baselines across different training shots.