Enhancing Vision-Language Models Generalization via Diversity-Driven Novel Feature Synthesis

TL;DR

This paper tackles the problem of CLIP’s diminished generalization when finetuned on downstream domains by introducing LDFS, a language-guided feature synthesis method. LDFS uses an instance-conditional local adaptation loss, stochastic text feature augmentation, and a pairwise regularizer to generate diverse, high-quality target-domain features from source-domain data, guided by language descriptions; these synthesized features are then used alongside original data to finetune CLIP. The approach is compatible with multiple CLIP finetuning strategies and does not require access to target-domain data, achieving improved generalization on unseen domains across several domain-shift benchmarks. The work demonstrates significant performance gains, robust ablations, and practical gains when integrated with prompting-based adaptation methods, highlighting its potential for real-world deployment where collecting target-domain data is impractical.

Abstract

Vision-language foundation models like CLIP have shown impressive zero-shot generalization, but finetuning on downstream datasets can cause overfitting and loss of its generalization ability on unseen domains. Although collecting additional data from new domains of interest is possible, this method is often impractical due to the challenges in obtaining annotated data. To address this, we propose a plug-and-play feature synthesis method called LDFS (Language-Guided Diverse Feature Synthesis) to synthesize new domain features and improve existing CLIP fine-tuning strategies. LDFS has three main contributions: 1) To synthesize novel domain features and promote diversity, we propose an instance-conditional feature augmentation strategy based on a text-guided feature augmentation loss. 2) To maintain feature quality after augmenting, we introduce a pairwise regularizer to preserve augmented feature coherence within the CLIP feature space. 3) We propose to use stochastic text feature augmentation to reduce the modality gap and further facilitate the process of text-guided feature synthesis. Extensive experiments show LDFS superiority in improving CLIP generalization ability on unseen domains without collecting data from those domains. The code will be made publicly available.

Figures (11)

• Figure 1: Our LDFS facilitates model adaptation to unseen domains using language description only, negating the need for direct data access from those new domains. Additionally, LDFS acts as a plug-and-play feature augmentation method, enhancing the efficacy of CLIP-based finetuning approaches in unseen domains.
• Figure 2: In Step 1 of our LDFS framework, a mapper network employs $\mathcal{L}_{ld}$ to translate source domain image features into diverse target domain features guided by a captioning model, ensuring instance-conditional text. Stochastic text feature augmentation is used to bridge the gap between image and text modalities, while a pairwise relation regularization loss $\mathcal{L}_{pair}$ aligns angles between original CLIP features and synthesized features, enhancing synthetic quality. In Step 2, CLIP is finetuned using both synthetic and original features for downstream adaptation through cross-entropy loss $\mathcal{L}_{con}.$
• Figure 3: Prior CLIP-based adaptation losses clipnadaladsclipstyler employ a consistent, global text description direction (i.e., $\Delta t$ in Fig. \ref{['f2']} (a)) for translating image features to the target domain. In contrast, our loss (Fig. \ref{['f2']} (b)) utilizes instance-specific local text description directions, enhancing the diversity of adapted image features.
• Figure 4: Comparison of nearest neighbor images of synthesized target domain features between LADS and our LDFS on the PACS dataset: photo → art, cartoon, and sketch. The first row displays randomly sampled images from the source domain (photo). Subsequent rows showcase the nearest neighbors of synthetic features from both LADS and LDFS. Red boxes highlight LADS' issues with homogeneous synthetic features, while green boxes indicate LADS' inability to preserve class information during adaptation.
• Figure 5: Style transfer photo images into art, cartoon, and sketch domains via CLIPStyler clipstyler. Unexpected artifacts (red box) and the model's inability to synthesize target objects (blue box) hinder its efficacy for downstream adaptation.