Federated Learning with Domain Shift Eraser

TL;DR

This paper tackles domain shift in federated learning by introducing Federated Domain Shift Eraser (FDSE), a decoupled framework that separates domain-agnostic feature extraction (DFE) from domain-specific skew erasing (DSE) via layer decomposition. FDSE enforces consistency across domains with a BN-based regularization term and aggregates shared components for consensus while personalizing DSE modules through similarity-aware aggregation. Empirical results on Office-Caltech10, PACS, and DomainNet show that FDSE achieves superior accuracy, robust convergence, and better generalization to unseen domains, while reducing communication and computation costs. The work demonstrates that a hybrid approach combining consensus enhancement with targeted personalization yields strong performance gains in cross-domain FL and offers a scalable path to broader applications and architectures.

Abstract

Federated learning (FL) is emerging as a promising technique for collaborative learning without local data leaving their devices. However, clients' data originating from diverse domains may degrade model performance due to domain shifts, preventing the model from learning consistent representation space. In this paper, we propose a novel FL framework, Federated Domain Shift Eraser (FDSE), to improve model performance by differently erasing each client's domain skew and enhancing their consensus. First, we formulate the model forward passing as an iterative deskewing process that extracts and then deskews features alternatively. This is efficiently achieved by decomposing each original layer in the neural network into a Domain-agnostic Feature Extractor (DFE) and a Domain-specific Skew Eraser (DSE). Then, a regularization term is applied to promise the effectiveness of feature deskewing by pulling local statistics of DSE's outputs close to the globally consistent ones. Finally, DFE modules are fairly aggregated and broadcast to all the clients to maximize their consensus, and DSE modules are personalized for each client via similarity-aware aggregation to erase their domain skew differently. Comprehensive experiments were conducted on three datasets to confirm the advantages of our method in terms of accuracy, efficiency, and generalizability.

Figures (8)

• Figure 1: Inllustration of the characteristic of the proposed FDSE v.s. Vallina FL mcmahan2017communication. FDSE decomposes the model to respectively erase domain skew for each client by $f_{DSE}^{(i)}$ and extract generalizable features for all the clients by $f_{DFE}$. This promotes consistency in representation space across domains since knowledge is fine-grainedly decoupled for learning and aggregation.
• Figure 2: The overview of the FDSE framework.
• Figure 3: Evaluation results on individual domains (i.e., clients) across the three datasets. Each axis represents the result for a specific domain and is scaled by the axis's maximum value for clarity.
• Figure 4: T-SNE visualization for representation space of different methods on Office-Caltech10, DomainNet, and PACS. Each color represents one class of samples and each shape represents one domain.
• Figure 5: Testing accuracy v.s. communication rounds.