Boosting Domain Incremental Learning: Selecting the Optimal Parameters is All You Need

TL;DR

The paper tackles the challenge of domain drift in deep models by enhancing Parameter-Isolation Domain Incremental Learning (PIDIL) with a trainable domain-label predictor called SOYO. SOYO combines a Gaussian Mixture Compressor (GMC) for memory-efficient storage of prior domain features, a Domain Feature Resampler (DFR) to balance training with pseudo-domain data, and a Multi-level Domain Feature Fusion Network (MDFN) to produce discriminative domain features from multi-layer backbone representations. The approach is validated across Domain Incremental Classification (DIC), Object Detection (DIOD), and Speech Enhancement (DISE) on six benchmarks, showing consistent gains in domain-prediction accuracy and downstream task metrics, often approaching oracle-style upper bounds and with manageable computational overhead. A key contribution is its broad compatibility with various Parameter-Efficient Fine-Tuning (PEFT) methods, enabling robust, memory-efficient adaptation in dynamic, real-world environments. Overall, SOYO provides a practical and scalable solution for accurate domain parameter selection in PIDIL, improving performance across multiple modalities and tasks.

Abstract

Deep neural networks (DNNs) often underperform in real-world, dynamic settings where data distributions change over time. Domain Incremental Learning (DIL) offers a solution by enabling continual model adaptation, with Parameter-Isolation DIL (PIDIL) emerging as a promising paradigm to reduce knowledge conflicts. However, existing PIDIL methods struggle with parameter selection accuracy, especially as the number of domains and corresponding classes grows. To address this, we propose SOYO, a lightweight framework that improves domain selection in PIDIL. SOYO introduces a Gaussian Mixture Compressor (GMC) and Domain Feature Resampler (DFR) to store and balance prior domain data efficiently, while a Multi-level Domain Feature Fusion Network (MDFN) enhances domain feature extraction. Our framework supports multiple Parameter-Efficient Fine-Tuning (PEFT) methods and is validated across tasks such as image classification, object detection, and speech enhancement. Experimental results on six benchmarks demonstrate SOYO's consistent superiority over existing baselines, showcasing its robustness and adaptability in complex, evolving environments. The codes will be released in https://github.com/qwangcv/SOYO.

Figures (5)

• Figure 1: Illustration of the proposed SOYO. (a) The green, blue, and red small squares represent learnable parameters for different domains. (b) The "pretrain" represents evaluation using the pre-trained model directly. The lower bound of the radar chart is set to 90% of the "pretrain", while the upper bound represents the achievable upper limit of the baseline if the domain labels of test samples are known in advance. Best viewed in color.
• Figure 2: Illustration of the proposed framework. The numbers 1, 2, 3, and 4 in (b) indicate the sequence of steps. Best viewed in color.
• Figure 3: BIC score and mAP on the Pascal VOC series dataset. In the right table, $K=0$ represents the baseline without GMC.
• Figure 4: Model training convergence curves on DomainNet. The black dashed arrow indicates that SOYO training is conducted on each domain (starting from the second domain) following the baseline training.
• Figure 5: t-SNE visualization of image features. Features are extracted by randomly selecting one image from each domain and class (345 classes across 6 domains) in the DomainNet dataset. (a) visualized the features extracted from the pre-trained ViT-B, and (b) shows the features $x^{D}$ in the MDFN module (see \ref{['fig:framework']}).