Federated Domain Generalization with Data-free On-server Matching Gradient

TL;DR

This work tackles Federated Domain Generalization by introducing FedOMG, a data-free on-server gradient matching method that leverages local client gradients to identify an invariant gradient direction across domains. By formulating a two-stage on-server optimization and an indirect IGD search via convex combinations of local gradients, FedOMG avoids second-order derivatives and data sharing, while integrating with existing FL baselines. Theoretical results bound the unseen-domain generalization gap to gradient divergence and domain divergence, and empirical results across MNIST, EMNIST, CIFAR, PACS, VLCS, and OfficeHome show consistent SOTA gains and robustness. Overall, FedOMG offers a practical, privacy-preserving approach to improve FL performance under domain shifts, with broad compatibility and improved generalization to unseen targets.

Abstract

Domain Generalization (DG) aims to learn from multiple known source domains a model that can generalize well to unknown target domains. One of the key approaches in DG is training an encoder which generates domain-invariant representations. However, this approach is not applicable in Federated Domain Generalization (FDG), where data from various domains are distributed across different clients. In this paper, we introduce a novel approach, dubbed Federated Learning via On-server Matching Gradient (FedOMG), which can \emph{efficiently leverage domain information from distributed domains}. Specifically, we utilize the local gradients as information about the distributed models to find an invariant gradient direction across all domains through gradient inner product maximization. The advantages are two-fold: 1) FedOMG can aggregate the characteristics of distributed models on the centralized server without incurring any additional communication cost, and 2) FedOMG is orthogonal to many existing FL/FDG methods, allowing for additional performance improvements by being seamlessly integrated with them. Extensive experimental evaluations on various settings to demonstrate the robustness of FedOMG compared to other FL/FDG baselines. Our method outperforms recent SOTA baselines on four FL benchmark datasets (MNIST, EMNIST, CIFAR-10, and CIFAR-100), and three FDG benchmark datasets (PACS, VLCS, and OfficeHome).

Figures (15)

• Figure 1: Illustrative toy task on two settings. 1) FL settings, where all users are participating in the training (left figure), 2) FDG setting: one user is excluded in the training (right figure).
• Figure 2: Performance comparison without pretrained models for $\alpha = 1, U = 20$.
• Figure 3: Global LR $\eta$.
• Figure 4: Searching ratio $\kappa$.
• Figure 5: The evaluations on gradient invariance. Algorithms with stronger invariance properties result in smaller gaps in cosine similarity between each domain's and the global gradients. The algorithms are evaluated on PACS dataset, where source domains are A, C, S, and target domain is P.

Theorems & Definitions (13)

• Definition 1: Invariant Gradient Direction 2022-DG-Fish
• Lemma 1: Triangle inequality for cosine similarity 2021-MF-TriangleCosine
• Definition 2: Pareto dominance 1999-OPT-Pareto
• Definition 3: Pareto optimality 1999-OPT-Pareto
• Lemma 2
• Theorem 1: FedOMG solution
• Corollary 1
• Lemma 3
• Theorem 2
• Corollary 2