Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Federated Unsupervised Domain Generalization using Global and Local Alignment of Gradients

Farhad Pourpanah, Mahdiyar Molahasani, Milad Soltany, Michael Greenspan, Ali Etemad

TL;DR

This work introduces federated unsupervised domain generalization, a setting where data privacy prohibits sharing raw data across clients. It proposes FedGaLA, a two-level gradient-alignment framework that performs local gradient alignment via self-supervised learning and global gradient alignment during aggregation to improve generalization to unseen domains. The authors establish a theoretical connection between domain shift and gradient covariance and demonstrate empirical gains across PACS, Office-Home, DomainNet, and TerraInc, along with thorough ablations and sensitivity analyses. The approach preserves privacy and is supported by public code, highlighting practical relevance for privacy-preserving distributed learning with domain shifts.

Abstract

We address the problem of federated domain generalization in an unsupervised setting for the first time. We first theoretically establish a connection between domain shift and alignment of gradients in unsupervised federated learning and show that aligning the gradients at both client and server levels can facilitate the generalization of the model to new (target) domains. Building on this insight, we propose a novel method named FedGaLA, which performs gradient alignment at the client level to encourage clients to learn domain-invariant features, as well as global gradient alignment at the server to obtain a more generalized aggregated model. To empirically evaluate our method, we perform various experiments on four commonly used multi-domain datasets, PACS, OfficeHome, DomainNet, and TerraInc. The results demonstrate the effectiveness of our method which outperforms comparable baselines. Ablation and sensitivity studies demonstrate the impact of different components and parameters in our approach. The source code is available at: https://github.com/MahdiyarMM/FedGaLA.

Federated Unsupervised Domain Generalization using Global and Local Alignment of Gradients

TL;DR

This work introduces federated unsupervised domain generalization, a setting where data privacy prohibits sharing raw data across clients. It proposes FedGaLA, a two-level gradient-alignment framework that performs local gradient alignment via self-supervised learning and global gradient alignment during aggregation to improve generalization to unseen domains. The authors establish a theoretical connection between domain shift and gradient covariance and demonstrate empirical gains across PACS, Office-Home, DomainNet, and TerraInc, along with thorough ablations and sensitivity analyses. The approach preserves privacy and is supported by public code, highlighting practical relevance for privacy-preserving distributed learning with domain shifts.

Abstract

We address the problem of federated domain generalization in an unsupervised setting for the first time. We first theoretically establish a connection between domain shift and alignment of gradients in unsupervised federated learning and show that aligning the gradients at both client and server levels can facilitate the generalization of the model to new (target) domains. Building on this insight, we propose a novel method named FedGaLA, which performs gradient alignment at the client level to encourage clients to learn domain-invariant features, as well as global gradient alignment at the server to obtain a more generalized aggregated model. To empirically evaluate our method, we perform various experiments on four commonly used multi-domain datasets, PACS, OfficeHome, DomainNet, and TerraInc. The results demonstrate the effectiveness of our method which outperforms comparable baselines. Ablation and sensitivity studies demonstrate the impact of different components and parameters in our approach. The source code is available at: https://github.com/MahdiyarMM/FedGaLA.
Paper Structure (29 sections, 6 theorems, 56 equations, 3 figures, 7 tables, 1 algorithm)

This paper contains 29 sections, 6 theorems, 56 equations, 3 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Approach
  4. Problem formulation
  5. Gradient alignment and domain shift
  6. FedGaLA
  7. Experiment setup
  8. Results
  9. Performance
  10. Empirical verification of Theorem \ref{['theorem']}
  11. Ablation studies and component analysis
  12. Hyperparameter sensitivity and stability
  13. Data and communication efficiency
  14. Model variations
  15. Conclusion
  16. ...and 14 more sections

Key Result

Theorem 1

Given Assumption assumption1, under the problem proposed in Definition def1, for two distinct domains characterized by random variables $\textbf{x}_i$ and $\textbf{x}_j$ belonging to two different clients $C_i$ and $C_j$, an increase in domain shift across the clients results in a decrease in covari

Figures (3)

  • Figure 1: Overview of FedGaLA.
  • Figure 2: (a) Covariance under domain shift across various domains of the PACS dataset. (b) Average ratio of discarded local gradients over 100 communication rounds ($\tau=0$ and $E=1$). (c) Impact of various batch sizes. (d) Impact of $\tau$ on performance across different number of local epochs. Results in (b-d) are reported for the domain $P$ of the PACS dataset, with a 10% label ratio.
  • Figure 3: (a) Impact of the number of iterations for global alignment. (b) Stability of the training. (c) Effect of different labeled data ratios on linear evaluation performance. (d) Performance of the model in different communication rounds. Results are reported for domain $P$ of PACS dataset with $10\%$ label ratio, except for (c) where the label ratio is changing.

Theorems & Definitions (19)

  • Definition 1
  • Theorem 1: Gradient Misalignment in Federated Self-supervised Learning Dependent upon Domain Shift
  • proof : Proof of Theorem 1
  • Lemma 1
  • Lemma 2
  • Claim 1
  • Corollary 1
  • Proposition 1
  • proof
  • proof
  • ...and 9 more