Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Domain Generalization using Pretrained Models without Fine-tuning

Ziyue Li, Kan Ren, Xinyang Jiang, Bo Li, Haipeng Zhang, Dongsheng Li

TL;DR

This work tackles domain generalization without finetuning by introducing SEDGE, which uses a shared label-space adapter applied to a pool of fixed pretrained models and a specialty-aware ensemble network to dispatch models for each sample. The key idea is that pretrained models show varying domain and class specialties, so dynamically selecting and combining them per instance yields better generalization while drastically reducing training cost. Empirical results on DomainBed demonstrate strong performance gains, especially when incorporating diverse pretraining (Pool-B), and reveal significant reductions in trainable parameters and training time. These findings offer a practical, scalable path to leveraging numerous pretrained models for DG in real-world deployments.

Abstract

Fine-tuning pretrained models is a common practice in domain generalization (DG) tasks. However, fine-tuning is usually computationally expensive due to the ever-growing size of pretrained models. More importantly, it may cause over-fitting on source domain and compromise their generalization ability as shown in recent works. Generally, pretrained models possess some level of generalization ability and can achieve decent performance regarding specific domains and samples. However, the generalization performance of pretrained models could vary significantly over different test domains even samples, which raises challenges for us to best leverage pretrained models in DG tasks. In this paper, we propose a novel domain generalization paradigm to better leverage various pretrained models, named specialized ensemble learning for domain generalization (SEDGE). It first trains a linear label space adapter upon fixed pretrained models, which transforms the outputs of the pretrained model to the label space of the target domain. Then, an ensemble network aware of model specialty is proposed to dynamically dispatch proper pretrained models to predict each test sample. Experimental studies on several benchmarks show that SEDGE achieves significant performance improvements comparing to strong baselines including state-of-the-art method in DG tasks and reduces the trainable parameters by ~99% and the training time by ~99.5%.

Domain Generalization using Pretrained Models without Fine-tuning

TL;DR

This work tackles domain generalization without finetuning by introducing SEDGE, which uses a shared label-space adapter applied to a pool of fixed pretrained models and a specialty-aware ensemble network to dispatch models for each sample. The key idea is that pretrained models show varying domain and class specialties, so dynamically selecting and combining them per instance yields better generalization while drastically reducing training cost. Empirical results on DomainBed demonstrate strong performance gains, especially when incorporating diverse pretraining (Pool-B), and reveal significant reductions in trainable parameters and training time. These findings offer a practical, scalable path to leveraging numerous pretrained models for DG in real-world deployments.

Abstract

Fine-tuning pretrained models is a common practice in domain generalization (DG) tasks. However, fine-tuning is usually computationally expensive due to the ever-growing size of pretrained models. More importantly, it may cause over-fitting on source domain and compromise their generalization ability as shown in recent works. Generally, pretrained models possess some level of generalization ability and can achieve decent performance regarding specific domains and samples. However, the generalization performance of pretrained models could vary significantly over different test domains even samples, which raises challenges for us to best leverage pretrained models in DG tasks. In this paper, we propose a novel domain generalization paradigm to better leverage various pretrained models, named specialized ensemble learning for domain generalization (SEDGE). It first trains a linear label space adapter upon fixed pretrained models, which transforms the outputs of the pretrained model to the label space of the target domain. Then, an ensemble network aware of model specialty is proposed to dynamically dispatch proper pretrained models to predict each test sample. Experimental studies on several benchmarks show that SEDGE achieves significant performance improvements comparing to strong baselines including state-of-the-art method in DG tasks and reduces the trainable parameters by ~99% and the training time by ~99.5%.
Paper Structure (18 sections, 6 equations, 5 figures, 3 tables)

This paper contains 18 sections, 6 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Domain Generalization
  4. Ensemble Learning
  5. Preliminaries
  6. Problem Formulation
  7. Preliminary Analysis
  8. SEDGE: A New Paradigm for Domain Generalization
  9. Framework: Pretrained Model without Fine-tuning
  10. Pretrained Model Pool
  11. Ensemble Network
  12. Learning Algorithm
  13. Experiments
  14. Evaluation Protocol
  15. DomainBed Benchmarking
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Different training paradigms in domain generalization.
  • Figure 2: The comparison of the average performance (x-axis, the higher the better) of different algorithms, their training time (y-axis, the smaller the better), and the number of their training parameters (the size of the marker). We also list the corresponding information (number of training parameters, test accuracy, training time) of each algorithm.
  • Figure 3: Performance distribution of the pretrained models over the samples within (a) different domains and (b) different classes. Each column of the left panel displays the relative performance distribution of the pretrained models; the right panel shows the Kullback-Leibler divergence between the performance distribution of different (a) domains and (b) classes. The comparison of the domain-level and class-level specialty shows that the performance of the pretrained models differs more significantly at the finer level.
  • Figure 4: SEDGE framework. Based on a pool of several fixed pretrained models, an ensemble network learns the matching of models and samples for model dispatching with the help of a label space adapter for prediction transformation.
  • Figure 5: Ranking models using the sum of ensemble weights on the sample in four domains of TerraIncognita. Each color block corresponds to a model. The higher rank indicates that this model is given a higher weight in predicting the samples in this domain.