Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Clients Collaborate: Flexible Differentially Private Federated Learning with Guaranteed Improvement of Utility-Privacy Trade-off

Yuecheng Li, Lele Fu, Tong Wang, Jian Lou, Bin Chen, Lei Yang, Jian Shen, Zibin Zheng, Chuan Chen

TL;DR

FedCEO addresses the utility-privacy tension in DP-enabled federated learning by enabling cross-client collaboration through a server-side tensor low-rank proximal update. By stacking noisy client updates into a third-order tensor and performing adaptive truncated tensor SVD (T-tSVD), the method smooths the global semantic space and recovers utility lost to DP noise. The authors prove a tighter utility-privacy bound of $\varepsilon_u \cdot \varepsilon_p \leq O\left(\frac{\sqrt{d}}{N}\right)$ and demonstrate empirical gains on EMNIST and CIFAR-10 across privacy settings, with robustness to gradient-leakage attacks via DP guarantees. This approach offers scalable DPFL with improved performance and suggests extensions to heterogeneous federated setups where semantic alignment across clients is advantageous.

Abstract

To defend against privacy leakage of user data, differential privacy is widely used in federated learning, but it is not free. The addition of noise randomly disrupts the semantic integrity of the model and this disturbance accumulates with increased communication rounds. In this paper, we introduce a novel federated learning framework with rigorous privacy guarantees, named FedCEO, designed to strike a trade-off between model utility and user privacy by letting clients ''Collaborate with Each Other''. Specifically, we perform efficient tensor low-rank proximal optimization on stacked local model parameters at the server, demonstrating its capability to flexibly truncate high-frequency components in spectral space. This capability implies that our FedCEO can effectively recover the disrupted semantic information by smoothing the global semantic space for different privacy settings and continuous training processes. Moreover, we improve the SOTA utility-privacy trade-off bound by order of $\sqrt{d}$, where $d$ is the input dimension. We illustrate our theoretical results with experiments on representative datasets and observe significant performance improvements and strict privacy guarantees under different privacy settings. The code is available at https://github.com/6lyc/FedCEO_Collaborate-with-Each-Other.

Clients Collaborate: Flexible Differentially Private Federated Learning with Guaranteed Improvement of Utility-Privacy Trade-off

TL;DR

FedCEO addresses the utility-privacy tension in DP-enabled federated learning by enabling cross-client collaboration through a server-side tensor low-rank proximal update. By stacking noisy client updates into a third-order tensor and performing adaptive truncated tensor SVD (T-tSVD), the method smooths the global semantic space and recovers utility lost to DP noise. The authors prove a tighter utility-privacy bound of and demonstrate empirical gains on EMNIST and CIFAR-10 across privacy settings, with robustness to gradient-leakage attacks via DP guarantees. This approach offers scalable DPFL with improved performance and suggests extensions to heterogeneous federated setups where semantic alignment across clients is advantageous.

Abstract

To defend against privacy leakage of user data, differential privacy is widely used in federated learning, but it is not free. The addition of noise randomly disrupts the semantic integrity of the model and this disturbance accumulates with increased communication rounds. In this paper, we introduce a novel federated learning framework with rigorous privacy guarantees, named FedCEO, designed to strike a trade-off between model utility and user privacy by letting clients ''Collaborate with Each Other''. Specifically, we perform efficient tensor low-rank proximal optimization on stacked local model parameters at the server, demonstrating its capability to flexibly truncate high-frequency components in spectral space. This capability implies that our FedCEO can effectively recover the disrupted semantic information by smoothing the global semantic space for different privacy settings and continuous training processes. Moreover, we improve the SOTA utility-privacy trade-off bound by order of , where is the input dimension. We illustrate our theoretical results with experiments on representative datasets and observe significant performance improvements and strict privacy guarantees under different privacy settings. The code is available at https://github.com/6lyc/FedCEO_Collaborate-with-Each-Other.
Paper Structure (41 sections, 7 theorems, 78 equations, 8 figures, 6 tables, 4 algorithms)

This paper contains 41 sections, 7 theorems, 78 equations, 8 figures, 6 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Federated Learning
  5. Differential Privacy
  6. Differential Privacy in Federated Learning
  7. Low-rankness
  8. Tensor Nuclear Norm
  9. T-product and Tensor SVD
  10. Main Algorithm
  11. FedCEO
  12. Analysis
  13. On the Scalability
  14. Utility-Privacy Trade-off Analysis
  15. Utility Analysis
  16. ...and 26 more sections

Key Result

Theorem 3.1

For each $\tau \geq 0$ and $\boldsymbol{\mathcal{W}_{\mathcal{N}}} \in \mathbb{R}^{d \times h \times K}$, our tensor low-rank proximal optimization objective defined in algorithm alg:alg2 obeys where $\operatorname{T-tSVD}(\cdot)$ is a truncated tSVD operator and $\frac{1}{2\tau}$ is the truncation threshold, defined as follows: where $\boldsymbol{\mathcal{D}}$ is an f-diagonal tensor whose each

Figures (8)

  • Figure 1: The heat map illustrates the smoothness of the global semantic space for ten clients at different training stages, where the color gradient from red to blue signifies an increase in smoothness. The bar chart illustrates the testing accuracy on the tenth class for each client during different training stages, where clients experiencing significant degradation in the semantic understanding of the tenth class are highlighted in pink, while others are marked in green. Due to the randomness of DP noise, we observe significant degradation in the local semantic representation of $10$-th class for clients 1, 3, and 6 among the ten clients (ACC significantly reduced), while the impact on other clients is relatively minor. Consequently, the corresponding blocks in the tenth row of the heat map matrix turn red, indicating a decrease in the smoothness of the global semantic space. In contrast, our approach enhances the smoothness of the global semantic space, evidenced by the recovery of the blue color in the tenth row of the heat map matrix. Simultaneously, the local semantic representation of class ten for clients 1, 3, and 6 is restored (ACC improved), based on the collaboration among all clients.
  • Figure 2: The visualization of the low-rank proximal optimization process, where $w_i$ is the $(i+1)$-th frequency component.
  • Figure 3: Privacy protection performance of three federated learning frameworks on CIFAR-10. Both FedCEO and UDP-FedAvg demonstrate robust defense against privacy attacks with smaller Peak Signal-to-Noise Ratio (PSNR), while DLG successfully infers sensitive images from clients in FedAvg.
  • Figure 4: Utility-Privacy Trade-off for our FedCEO and other methods on CIFAR-10.
  • Figure 5: The singular value curves in spectral space for the original noise parameter tensors on two real datasets. $\omega_0$ to $\omega_9$ represent components from low to high frequency.
  • ...and 3 more figures

Theorems & Definitions (23)

  • Definition 2.1: Differential Privacy dwork2006DPDPDL
  • Definition 2.2: Tensor Nuclear Norm lu2016TNN
  • Definition 2.3: Tensor Singular Value Decomposition kilmer2011T-tSVDlu2019TRPCA
  • Theorem 3.1: Interpretability
  • proof
  • Proposition 3.2
  • proof
  • Definition 4.1: Utility Loss zhang2022utilityDef
  • Definition 4.2: Model Utility
  • Theorem 4.3: Utility Analysis of FedCEO
  • ...and 13 more