Semantic Relation-Enhanced CLIP Adapter for Domain Adaptive Zero-Shot Learning

TL;DR

The paper tackles domain-adaptive zero-shot learning (DAZSL) with CLIP by introducing the Semantic Relation-Enhanced CLIP (SRE-CLIP) Adapter. It couples an image-encoder adapter with WordNet-guided, GCN-enhanced class prototypes and adds two targeted losses—Semantic Relation Structure Loss $\mathcal{L}_{srs}$ and Cross-Modal Alignment Retention $\mathcal{L}_{align}$—to enable efficient cross-domain transfer while preserving zero-shot cross-modal alignment. Through a two-stage training regime combining source-domain supervision and target-domain information maximization, SRE-CLIP achieves state-of-the-art results on I2AwA and I2WebV, with substantial improvements in unseen-class recognition and overall H-scores. The approach demonstrates the practical impact of integrating structured semantic relations into vision-language adapters for robust DAZSL performance, and provides code for reproducibility.

Abstract

The high cost of data annotation has spurred research on training deep learning models in data-limited scenarios. Existing paradigms, however, fail to balance cross-domain transfer and cross-category generalization, giving rise to the demand for Domain-Adaptive Zero-Shot Learning (DAZSL). Although vision-language models (e.g., CLIP) have inherent advantages in the DAZSL field, current studies do not fully exploit their potential. Applying CLIP to DAZSL faces two core challenges: inefficient cross-category knowledge transfer due to the lack of semantic relation guidance, and degraded cross-modal alignment during target domain fine-tuning. To address these issues, we propose a Semantic Relation-Enhanced CLIP (SRE-CLIP) Adapter framework, integrating a Semantic Relation Structure Loss and a Cross-Modal Alignment Retention Strategy. As the first CLIP-based DAZSL method, SRE-CLIP achieves state-of-the-art performance on the I2AwA and I2WebV benchmarks, significantly outperforming existing approaches.

Figures (2)

• Figure 1: This figure illustrates the SRE-CLIP Adapter framework (a), which integrates cross-modal alignment and semantic relation learning to enhance knowledge transfer across domains. Additionally, we propose two strategies, Semantic Relation Structure Loss (b,c) and Cross-modal Alignment Retention Strategy (d), to facilitate efficient knowledge transfer while preserving the VLM zero-shot recognition capability.
• Figure 2: Visualization of image feature via t-SNE. Each point among them represents a sample, and different colors represent its different categories.