Adapting In-Domain Few-Shot Segmentation to New Domains without Source Domain Retraining

TL;DR

This work tackles cross-domain few-shot segmentation by proposing Informative Structure Adaptation (ISA), a framework that enables existing in-domain FSS models to adapt to unseen domains during inference without retraining on source data. ISA consists of Informative Structure Identification (ISI), which uses a data-dependent structure Fisher score built on the diagonal empirical Fisher information to select a single informative backbone layer for adaptation, and Progressive Structure Adaptation (PSA), which trains with hierarchically constructed support samples from 1 to K−1 shots to progressively mitigate domain shifts. Together, ISI and PSA form a model-agnostic pipeline that can retrofit both CNN and transformer-based FSS methods, yielding substantial CD-FSS gains across multiple benchmarks while avoiding source-domain retraining. The approach is complemented by discussions on Fisher Information theory, supports fast ISA variants, and demonstrates practical benefits for real-world deployment where labeled target-domain data are scarce.

Abstract

Cross-domain few-shot segmentation (CD-FSS) aims to segment objects of novel classes in new domains, which is often challenging due to the diverse characteristics of target domains and the limited availability of support data. Most CD-FSS methods redesign and retrain in-domain FSS models using abundant base data from the source domain, which are effective but costly to train. To address these issues, we propose adapting informative model structures of the well-trained FSS model for target domains by learning domain characteristics from few-shot labeled support samples during inference, thereby eliminating the need for source domain retraining. Specifically, we first adaptively identify domain-specific model structures by measuring parameter importance using a novel structure Fisher score in a data-dependent manner. Then, we progressively train the selected informative model structures with hierarchically constructed training samples, progressing from fewer to more support shots. The resulting Informative Structure Adaptation (ISA) method effectively addresses domain shifts and equips existing well-trained in-domain FSS models with flexible adaptation capabilities for new domains, eliminating the need to redesign or retrain CD-FSS models on base data. Extensive experiments validate the effectiveness of our method, demonstrating superior performance across multiple CD-FSS benchmarks. Codes are at https://github.com/fanq15/ISA.

Figures (4)

• Figure 1: Cross-domain few-shot segmentation is usually challenging due to ➀ diverse characteristics of target domains (e.g., requiring high- or low-level parsing) and the ➁ limited support data (e.g., 1 to 5 shots). Our Informative Structure Adaptation (ISA) can adaptively identify and efficiently train domain-specific informative structures for the target domains during inference. This process is applied to few-shot annotated support data during inference and can be directly integrated with various well-trained FSS methods fan2022selfwang2019panetzhang2022featurezhang2023personalizesu2024domainwithout source domain retraining, leading to substantial improvements in CD-FSS.
• Figure 2: Qualitative comparisons between our method and the baseline model in the 1-way 5-shot setting across four target domain datasets. We show only one support image for clarity.
• Figure 3: Selected trainable layer distribution of the informative structure identification (ISI) module. The “selected ratio” denotes the frequency with which each layer is selected across the entire dataset.
• Figure 4: Comparisons on training loss, testing loss and mIoU for various model training strategies.