Diverse Target and Contribution Scheduling for Domain Generalization

TL;DR

This work tackles domain generalization under distribution shift by revealing gradient conflicts caused by uniform one-hot targets and equal-source-domain weights. It introduces Diverse Target and Contribution Scheduling (DTCS), decomposed into Diverse Target Supervision (DTS) and Diverse Contribution Balance (DCB), to partition optimization into non-conflicting sub-tasks and to balance domain influences adaptively. Theoretical analysis links gradient conflicts to empirical source risk and distribution gaps, while extensive experiments on PACS, Office-Home, Terra-Incognita, and VLCS show DTCS achieves competitive or state-of-the-art performance, particularly on hard-to-transfer domains. The approach is shown to be broadly compatible with existing DG methods, illustrating its practical impact for robust domain generalization in vision tasks.

Abstract

Generalization under the distribution shift has been a great challenge in computer vision. The prevailing practice of directly employing the one-hot labels as the training targets in domain generalization~(DG) can lead to gradient conflicts, making it insufficient for capturing the intrinsic class characteristics and hard to increase the intra-class variation. Besides, existing methods in DG mostly overlook the distinct contributions of source (seen) domains, resulting in uneven learning from these domains. To address these issues, we firstly present a theoretical and empirical analysis of the existence of gradient conflicts in DG, unveiling the previously unexplored relationship between distribution shifts and gradient conflicts during the optimization process. In this paper, we present a novel perspective of DG from the empirical source domain's risk and propose a new paradigm for DG called Diverse Target and Contribution Scheduling (DTCS). DTCS comprises two innovative modules: Diverse Target Supervision (DTS) and Diverse Contribution Balance (DCB), with the aim of addressing the limitations associated with the common utilization of one-hot labels and equal contributions for source domains in DG. In specific, DTS employs distinct soft labels as training targets to account for various feature distributions across domains and thereby mitigates the gradient conflicts, and DCB dynamically balances the contributions of source domains by ensuring a fair decline in losses of different source domains. Extensive experiments with analysis on four benchmark datasets show that the proposed method achieves a competitive performance in comparison with the state-of-the-art approaches, demonstrating the effectiveness and advantages of the proposed DTCS.

Figures (8)

• Figure 1: The origin of gradient conflicts in DG and our mitigation strategy.$\theta_1^*$, $\theta_2^*$, $\theta_3^*$ denote the optimal solutions in different source domains, and $\theta$ represents the current model parameter. (a) In DG, the probability density functions exhibit variations across domains, resulting in a multi-modal distribution for one class and thus causing disparate gradient descent directions across domains. The gradients in different domains may contradict with each other, hindering the optimization process. (b) Introducing diverse targets for source domains experiencing distribution shifts, our proposed strategy aims to implicitly divide the overall optimization process into distinct sub-tasks that do not interface each other.
• Figure 2: The training losses of different source domains for ERM with the pre-trained ResNet-18 as the backbone and 'Photo' as the target domain, source domain 1, 2, and 3 denote 'Art painting', 'Cartoon', and 'Sketch', respectively.
• Figure 3: The softmax logits (the top-3 predicted classes and their corresponding confidences) generated by the specialist models trained in 'Art', 'Cartoon', 'Photo', and 'Sketch', respectively. The dog's picture is sampled from 'Art'.
• Figure 4: Illustration of the proposed Diverse Target and Contribution Scheduling (DTCS). We propose three distinct paradigms for DTCS, with diverse target prophets ranging from complex to simple, while ensuring negligible performance degradation.
• Figure 5: The illustration of our proposed Diverse Target and Contribution Scheduling (DTCS). (a) Addition to the one-hot labels, we also utilize the soft labels to surrogate the diverse targets, dividing the overall task into non-contradictory sub-tasks. Besides, we balance the contributions of different source domains via re-weighting their losses according to the relative inverse training rate every iteration instead of directly summing up or averaging the losses across source domains. (b) We provide four variants of diverse target supervision, with diverse target prophets ranging from complex models to simple models, from pre-trained models to progressive learning models, and from models with a large number of parameters to those with a reduced set.

Theorems & Definitions (5)

• Theorem 1
• Definition 1
• Lemma 1
• Proof A.1
• Proof A.2