Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Privacy-preserving Federated Primal-dual Learning for Non-convex and Non-smooth Problems with Model Sparsification

Yiwei Li, Chien-Wei Huang, Shuai Wang, Chong-Yung Chi, Tony Q. S. Quek

TL;DR

The paper tackles privacy-preserving federated learning for non-convex and non-smooth objectives under communication constraints. It introduces two DP-based primal-dual FL algorithms: DP-FedPDM and BSDP-FedPDM, with bidirectional model sparsification to reduce signaling overhead while maintaining privacy guarantees. The authors provide privacy analyses showing how per-round and total privacy losses scale with participation and local updates, and a convergence proof establishing an $\mathcal{O}(1/\zeta)$ round complexity to reach a $\zeta$-stationary solution. Empirical results on MNIST and Adult data demonstrate superior performance and robustness to DP noise compared to state-of-the-art baselines, highlighting practical impact for efficient and private FL in IoT and edge systems.

Abstract

Federated learning (FL) has been recognized as a rapidly growing research area, where the model is trained over massively distributed clients under the orchestration of a parameter server (PS) without sharing clients' data. This paper delves into a class of federated problems characterized by non-convex and non-smooth loss functions, that are prevalent in FL applications but challenging to handle due to their intricate non-convexity and non-smoothness nature and the conflicting requirements on communication efficiency and privacy protection. In this paper, we propose a novel federated primal-dual algorithm with bidirectional model sparsification tailored for non-convex and non-smooth FL problems, and differential privacy is applied for privacy guarantee. Its unique insightful properties and some privacy and convergence analyses are also presented as the FL algorithm design guidelines. Extensive experiments on real-world data are conducted to demonstrate the effectiveness of the proposed algorithm and much superior performance than some state-of-the-art FL algorithms, together with the validation of all the analytical results and properties.

Privacy-preserving Federated Primal-dual Learning for Non-convex and Non-smooth Problems with Model Sparsification

TL;DR

The paper tackles privacy-preserving federated learning for non-convex and non-smooth objectives under communication constraints. It introduces two DP-based primal-dual FL algorithms: DP-FedPDM and BSDP-FedPDM, with bidirectional model sparsification to reduce signaling overhead while maintaining privacy guarantees. The authors provide privacy analyses showing how per-round and total privacy losses scale with participation and local updates, and a convergence proof establishing an round complexity to reach a -stationary solution. Empirical results on MNIST and Adult data demonstrate superior performance and robustness to DP noise compared to state-of-the-art baselines, highlighting practical impact for efficient and private FL in IoT and edge systems.

Abstract

Federated learning (FL) has been recognized as a rapidly growing research area, where the model is trained over massively distributed clients under the orchestration of a parameter server (PS) without sharing clients' data. This paper delves into a class of federated problems characterized by non-convex and non-smooth loss functions, that are prevalent in FL applications but challenging to handle due to their intricate non-convexity and non-smoothness nature and the conflicting requirements on communication efficiency and privacy protection. In this paper, we propose a novel federated primal-dual algorithm with bidirectional model sparsification tailored for non-convex and non-smooth FL problems, and differential privacy is applied for privacy guarantee. Its unique insightful properties and some privacy and convergence analyses are also presented as the FL algorithm design guidelines. Extensive experiments on real-world data are conducted to demonstrate the effectiveness of the proposed algorithm and much superior performance than some state-of-the-art FL algorithms, together with the validation of all the analytical results and properties.
Paper Structure (32 sections, 6 theorems, 55 equations, 8 figures, 1 table, 2 algorithms)

This paper contains 32 sections, 6 theorems, 55 equations, 8 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Contributions
  4. Preliminaries
  5. Federated Learning
  6. Differential Privacy
  7. Proposed Privacy-preserving Federated Primal-dual Method
  8. Proposed Primal-dual Method in FL
  9. DP-FedPDM with Bidirectional Model Sparsification
  10. Privacy and Convergence Analysis for DP-FedPDM
  11. Assumptions
  12. Privacy Analysis
  13. Convergence Analysis
  14. Experimental Results and Discussions
  15. Experimental Model
  16. ...and 17 more sections

Key Result

Lemma 1

dwork2014algorithmic Suppose that the randomized mechanism ${\cal M}$ satisfies $(\epsilon,\delta)$-DP defined in eqn: DP gaussian. Then, the minimum required noise variance $\sigma^2$ is given by where $s$, the $\ell_2$-norm sensitivity of ${\bm g}$ in eqn: DP gaussian, is given by in which ${\cal X}$ is the domain of function ${\bm g}$.

Figures (8)

  • Figure 1: The framework of vanilla FL system.
  • Figure 2: Testing accuracy and uplink communication costs of the proposed BSDP-FedPDM for $\alpha_{_D} = 1$ and $\alpha_{_U} \in \{0.05, 0.1, 0.3, 0.5, 0.7, 0.9, 1 \}$.
  • Figure 3: Testing accuracy of the proposed BSDP-FedPDM for $\alpha_{_D} \in \{0.5, 0.75, 0.9, 1 \}$ and $\alpha_{_U} = 0.5$.
  • Figure 4: Performance of the proposed DP-FedPDM on (a) Adult dataset for the cases of "without DP", $\bar{\epsilon}_i^T \in \{0.2,0.5,1\}$ and (b) MNIST dataset for the cases of "without DP", $\bar{\epsilon}_i^T \in \{10,20,30\}$.
  • Figure 5: Performance of DP-FedPDM for $\beta \in \{0, 0.05, 0.5, 5\}$ for (a) Adult dataset, where $\bar{\epsilon}_i^T = 0.5$, and (b) MNIST dataset, where $\bar{\epsilon}_i^T = 20$.
  • ...and 3 more figures

Theorems & Definitions (11)

  • Definition 1
  • Lemma 1
  • Definition 2
  • Definition 3: $\mathop{\mathrm{top}}\nolimits_k$ and $\mathop{\mathrm{rand}}\nolimits_k$ Sparsifiers hu2022federated
  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Remark 1
  • Remark 2
  • Lemma 2
  • ...and 1 more