Adaptive Semantic Consistency for Cross-domain Few-shot Classification

TL;DR

This work addresses cross-domain few-shot classification (CD-FSC), where domain shift and scarce target data lead to overfitting when finetuning pre-trained models. It introduces Adaptive Semantic Consistency (ASC), a plug-and-play framework that preserves source-domain knowledge by reweighting source samples according to their similarity to the target domain and by enforcing semantic consistency between a fixed source model and a tunable target model through an auxiliary copy during finetuning. ASC is designed to be compatible with multiple CD-FSC baselines (Finetune, ConFT, SupCon) and adds a semantic-consistency loss $L_{con}$ to the standard task loss, yielding $L_{ASC} = L_{cls-F} + \lambda L_{con}$. Empirical results on eight target datasets show that ASC consistently improves cross-domain performance, demonstrating its effectiveness in mitigating overfitting and preserving transferable knowledge, with code released for reproducibility.

Abstract

Cross-domain few-shot classification (CD-FSC) aims to identify novel target classes with a few samples, assuming that there exists a domain shift between source and target domains. Existing state-of-the-art practices typically pre-train on source domain and then finetune on the few-shot target data to yield task-adaptive representations. Despite promising progress, these methods are prone to overfitting the limited target distribution since data-scarcity and ignore the transferable knowledge learned in the source domain. To alleviate this problem, we propose a simple plug-and-play Adaptive Semantic Consistency (ASC) framework, which improves cross-domain robustness by preserving source transfer capability during the finetuning stage. Concretely, we reuse the source images in the pretraining phase and design an adaptive weight assignment strategy to highlight the samples similar to target domain, aiming to aggregate informative target-related knowledge from source domain. Subsequently, a semantic consistency regularization is applied to constrain the consistency between the semantic features of the source images output by the source model and target model. In this way, the proposed ASC enables explicit transfer of source domain knowledge to prevent the model from overfitting the target domain. Extensive experiments on multiple benchmarks demonstrate the effectiveness of the proposed ASC, and ASC provides consistent improvements over the baselines. The source code is released at https://github.com/luhc666/ASC-CDFSL.

Figures (6)

• Figure 1: An illustration of prior learned target-related knowledge. The source images used in pre-train phase contain similar semantic features as target images. The capability of extracting such transferable features should be preserved in finetuning phase.
• Figure 2: Overview of our Adaptive Semantic Consistency (ASC) framework. Firstly, we measure the similarity between each source sample and target domain by their euclidean distance and reweight the source images based on the source-target similarity. After that, we compute the consistency loss $L_{con}$ by measuring the distance between the output of source and target model on the same images and jointly minimize $L_{con}$ with classification loss $L_{cls-F}$.
• Figure 3: Parameters change of each block after finetuning on 4 target datasets regularizing different levels of features output by different blocks of the backbone, which is denoted by different colors of lines in each subfigure.
• Figure 4: 5-way 5/1-shot results of Supcon + ASC with either source images or target images for semantic regularization.
• Figure 5: Parameters change of each block after finetuning on 4 target datasets with target images to conduct semantic consistency regularization.