Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Eliminating Domain Bias for Federated Learning in Representation Space

Jianqing Zhang, Yang Hua, Jian Cao, Hao Wang, Tao Song, Zhengui Xue, Ruhui Ma, Haibing Guan

TL;DR

This work tackles representation bias and degeneration in federated learning under statistical heterogeneity by introducing Domain Bias Eliminator (DBE), composed of Personalized Representation Bias Memory (PRBM) and Mean Regularization (MR). PRBM stores client-specific bias while DBE creates a global mean translation to guide learning, enabling bi-directional knowledge transfer between server and clients. The authors provide theoretical bounds showing reduced domain discrepancy improves both local-to-global and global-to-local transfer, and they validate DBE across CV and NLP tasks, where FedAvg+DBE achieves substantial generalization and personalization gains and outperforms many state-of-the-art personalized FL methods. The approach preserves privacy and incurs negligible overhead, while demonstrating broad applicability to existing FL methods and real-world scenarios.

Abstract

Recently, federated learning (FL) is popular for its privacy-preserving and collaborative learning abilities. However, under statistically heterogeneous scenarios, we observe that biased data domains on clients cause a representation bias phenomenon and further degenerate generic representations during local training, i.e., the representation degeneration phenomenon. To address these issues, we propose a general framework Domain Bias Eliminator (DBE) for FL. Our theoretical analysis reveals that DBE can promote bi-directional knowledge transfer between server and client, as it reduces the domain discrepancy between server and client in representation space. Besides, extensive experiments on four datasets show that DBE can greatly improve existing FL methods in both generalization and personalization abilities. The DBE-equipped FL method can outperform ten state-of-the-art personalized FL methods by a large margin. Our code is public at https://github.com/TsingZ0/DBE.

Eliminating Domain Bias for Federated Learning in Representation Space

TL;DR

This work tackles representation bias and degeneration in federated learning under statistical heterogeneity by introducing Domain Bias Eliminator (DBE), composed of Personalized Representation Bias Memory (PRBM) and Mean Regularization (MR). PRBM stores client-specific bias while DBE creates a global mean translation to guide learning, enabling bi-directional knowledge transfer between server and clients. The authors provide theoretical bounds showing reduced domain discrepancy improves both local-to-global and global-to-local transfer, and they validate DBE across CV and NLP tasks, where FedAvg+DBE achieves substantial generalization and personalization gains and outperforms many state-of-the-art personalized FL methods. The approach preserves privacy and incurs negligible overhead, while demonstrating broad applicability to existing FL methods and real-world scenarios.

Abstract

Recently, federated learning (FL) is popular for its privacy-preserving and collaborative learning abilities. However, under statistically heterogeneous scenarios, we observe that biased data domains on clients cause a representation bias phenomenon and further degenerate generic representations during local training, i.e., the representation degeneration phenomenon. To address these issues, we propose a general framework Domain Bias Eliminator (DBE) for FL. Our theoretical analysis reveals that DBE can promote bi-directional knowledge transfer between server and client, as it reduces the domain discrepancy between server and client in representation space. Besides, extensive experiments on four datasets show that DBE can greatly improve existing FL methods in both generalization and personalization abilities. The DBE-equipped FL method can outperform ten state-of-the-art personalized FL methods by a large margin. Our code is public at https://github.com/TsingZ0/DBE.
Paper Structure (47 sections, 7 theorems, 27 equations, 9 figures, 9 tables, 1 algorithm)

This paper contains 47 sections, 7 theorems, 27 equations, 9 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Notations and Preliminaries
  4. Notations
  5. Traditional Federated Learning
  6. Method
  7. Problem Statement
  8. Personalized Representation Bias Memory (PRBM)
  9. Mean Regularization (MR)
  10. Improved Bi-directional Knowledge Transfer
  11. Local-To-Global Knowledge Transfer
  12. Global-To-Local Knowledge Transfer
  13. Negligible Additional Communication and Computation Overhead
  14. Privacy-Preserving Discussion
  15. Experiments
  16. ...and 32 more sections

Key Result

Corollary 1

Consider a local data domain $\mathcal{D}_i$ and a virtual global data domain $\mathcal{D}$ for client $i$ and the server, respectively. Let $\mathcal{D}_i = \langle \mathcal{U}_i, c^* \rangle$ and $\mathcal{D} = \langle \mathcal{U}, c^* \rangle$, where $c^*: \mathcal{X} \mapsto \mathcal{Y}$ is a gr where $\mathcal{L}_{\hat{\mathcal{D}}_i}$ is the empirical loss on $\mathcal{D}_i$, $e$ is the base

Figures (9)

  • Figure 1: t-SNE van2008visualizing visualization and per-layer MDL (bits) for representations before/after local training in FedAvg. We use color and shape to distinguish labels and clients respectively for t-SNE. A large MDL means low representation quality. Best viewed in color and zoom-in.
  • Figure 2: The illustration of the local model. We emphasize the parts that correspond to PRBM and MR with red and green, respectively.
  • Figure 3: t-SNE visualization for representations on Tiny-ImageNet (200 labels). "B" and "A" denote "before local training" and "after local training", respectively. We use color and shape to distinguish labels and clients, respectively. Best viewed in color and zoom-in.
  • Figure 4: The training loss and test accuracy curve of FedAvg+DBE on FMNIST dataset using the 4-layer CNN in the practical setting.
  • Figure 5: t-SNE visualization for the representations extracted by the global feature extractor on AG News (four labels) in FedAvg+$\texttt{DBE}$. We use color and shape to distinguish labels and clients, respectively.
  • ...and 4 more figures

Theorems & Definitions (10)

  • Corollary 1
  • Corollary 2
  • Definition 1
  • Theorem 1
  • Theorem 2
  • Corollary 1
  • proof
  • Theorem 3
  • Corollary 2
  • proof