Enhancing Information Maximization with Distance-Aware Contrastive Learning for Source-Free Cross-Domain Few-Shot Learning

TL;DR

This paper explores a Source-Free CDFSL (SF-CDFSL) problem, in which CDFSL is addressed through the use of existing pretrained models instead of training a model with source data, avoiding accessing source data.

Abstract

Existing Cross-Domain Few-Shot Learning (CDFSL) methods require access to source domain data to train a model in the pre-training phase. However, due to increasing concerns about data privacy and the desire to reduce data transmission and training costs, it is necessary to develop a CDFSL solution without accessing source data. For this reason, this paper explores a Source-Free CDFSL (SF-CDFSL) problem, in which CDFSL is addressed through the use of existing pretrained models instead of training a model with source data, avoiding accessing source data. This paper proposes an Enhanced Information Maximization with Distance-Aware Contrastive Learning (IM-DCL) method to address these challenges. Firstly, we introduce the transductive mechanism for learning the query set. Secondly, information maximization (IM) is explored to map target samples into both individual certainty and global diversity predictions, helping the source model better fit the target data distribution. However, IM fails to learn the decision boundary of the target task. This motivates us to introduce a novel approach called Distance-Aware Contrastive Learning (DCL), in which we consider the entire feature set as both positive and negative sets, akin to Schrodinger's concept of a dual state. Instead of a rigid separation between positive and negative sets, we employ a weighted distance calculation among features to establish a soft classification of the positive and negative sets for the entire feature set. Furthermore, we address issues related to IM by incorporating contrastive constraints between object features and their corresponding positive and negative sets. Evaluations of the 4 datasets in the BSCD-FSL benchmark indicate that the proposed IM-DCL, without accessing the source domain, demonstrates superiority over existing methods, especially in the distant domain task.

Figures (11)

• Figure 1: Contrast the proposed SF-CDFSL problem with existing closely related problem settings. (a) Vanilla UDA: Both source and target tasks share the same label space, but the data of these two domains are from different distributions. (b) SFDA: the source data are not accessible in UDA. Contrasted with the DA problem, (c) Vanilla CDFSL: including sufficient labeled source data and a few labeled target data (there are both domain gaps and task shift between source and target task). While (d) Our proposed SF-CDFSL: in which the source data is not accessible (like SFDA) and contains both domain gaps and task shift (like CDFSL), is more challenging. In SF-CDFSL, we solve the FSL problem in the target domain with limited target data and a pretrained source model without accessing any source data. This effectively eliminates the potential information leakage from the source domain. The lock means that the corresponding part is not accessible when retraining the target task, accessible only for pre-training.
• Figure 2: Overview of the proposed IM-DCL method. The images from the support set (with labels) and query set (without labels) are forwarded to the source model to obtain corresponding feature representations. In the supervised inductive process, the support set samples are optimized through supervised CE loss and IM loss. While during transductive process, the proposed DCL was used to enhance IM loss.
• Figure 3: IM cannot tell whether a sample is correctly classified or not. (a) represents the categorical objectives of the samples in the target domain, i.e., the query set is correctly classified. However, problem (b) may arise if IM operates on CDFSL, e.g., if the lion and cat in the query set are classified into each other's category, IM cannot tell this case. Therefore, an DCL algorithm is explored to solve this problem.
• Figure 4: Distance-aware Contrastive Learning algorithm (DCL) achieves contrastive learning by maximizing the similarity of a feature to its positive set while minimizing the similarity to its negative set. (a) illustrates the operation manner of DCL, in which the positive weight matrix are obtained from the distances between all the features and the object feature $\textbf{f}_{t}$. The negative weight matrix is obtained through the positive weight matrix. (b) illustrates the three ways about getting the negative weight matrix from the positive one.
• Figure 5: Visualization results on EuroSAT and ISIC. Left part means the results on all-way task. Right part shows the visualization on 5-way task. Different color means different categories. Source model means that the result when the source model is not fine-tuned.