DSP-Reg: Domain-Sensitive Parameter Regularization for Robust Domain Generalization

TL;DR

This work tackles domain generalization by shifting focus from feature-level invariance to parameter-level sensitivity. It introduces a covariance-based framework to quantify per-parameter domain sensitivity via gradient propagation, deriving a per-parameter sensitivity index $s_k$ and cross-domain coefficient of variation $c_k$. Based on this, DSP-Reg softly regularizes domain-sensitive parameters through $\\\\\\\\\nR_{DS} = \sum_k c_k (\\partial_{\\theta_k} \\mathcal{L}_{sup})^2$, updating $c_k$ periodically to emphasize domain-invariant learning. Extensive experiments on five DG benchmarks show state-of-the-art or competitive performance with modest computational overhead, highlighting the value of parameter-level regularization in improving unseen-domain generalization. The method is complementary to existing DG approaches and offers a principled bridge between Fisher information, Bayesian uncertainty, and domain-consistency enforcement.

Abstract

Domain Generalization (DG) is a critical area that focuses on developing models capable of performing well on data from unseen distributions, which is essential for real-world applications. Existing approaches primarily concentrate on learning domain-invariant features, which assume that a model robust to variations in the source domains will generalize well to unseen target domains. However, these approaches neglect a deeper analysis at the parameter level, which makes the model hard to explicitly differentiate between parameters sensitive to domain shifts and those robust, potentially hindering its overall ability to generalize. In order to address these limitations, we first build a covariance-based parameter sensitivity analysis framework to quantify the sensitivity of each parameter in a model to domain shifts. By computing the covariance of parameter gradients across multiple source domains, we can identify parameters that are more susceptible to domain variations, which serves as our theoretical foundation. Based on this, we propose Domain-Sensitive Parameter Regularization (DSP-Reg), a principled framework that guides model optimization by a soft regularization technique that encourages the model to rely more on domain-invariant parameters while suppressing those that are domain-specific. This approach provides a more granular control over the model's learning process, leading to improved robustness and generalization to unseen domains. Extensive experiments on benchmarks, such as PACS, VLCS, OfficeHome, and DomainNet, demonstrate that DSP-Reg outperforms state-of-the-art approaches, achieving an average accuracy of 66.7\% and surpassing all baselines.

Figures (2)

• Figure 1: Visualization of parameter sensitivities $c_k$ for 2500 randomly sampled parameters from the 13th layer of ResNet-50. (a) Initial $c_k$ values computed at the beginning of training; (b) $c_k$ values after completion of training. Brighter colors indicate higher domain sensitivity.
• Figure 2: (a) Impact of the update frequency of sensitivity coefficient. Average accuracy on PACS dataset for different values of $T_{\mathrm{update}}$. Updating every 2 epochs provides the optimal trade-off between performance and computational cost. (b) Regularization strength analysis. Average accuracy on PACS dataset as a function of the regularization parameter $\lambda$. The optimal value $\lambda = 0.001$ balances domain-invariance with supervised learning.

Theorems & Definitions (11)

• Theorem 1
• proof
• Theorem 2
• proof
• Definition 1: Domain-invariant dependence
• Lemma 1: Invariant dependence $\Rightarrow$ constant sensitivity
• proof
• Lemma 2: Sensitivity disparity $\Leftrightarrow$ Jacobian-energy disparity
• proof
• Proposition 1: Contrapositive certificate of domain-specificity