Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sparse Decentralized Federated Learning

Shan Sha, Shenglong Zhou, Lingchen Kong, Geoffrey Ye Li

TL;DR

This work tackles the efficiency, stability, and privacy challenges of decentralized federated learning (DFL) by introducing Sparse DFL (SDFL) and the CEPS algorithm. CEPS combines a sparsity constraint with 1-bit compressive sensing (1BCS) to drastically reduce communication, while employing inexact ADMM to enable efficient, distributed optimization with partial neighbor participation. Privacy is guaranteed via the Gaussian mechanism, providing $(\\varepsilon,\\delta)$-DP per iteration and cumulatively through adaptive composition, along with theoretical convergence guarantees under standard assumptions. Numerical experiments on linear and logistic regression tasks demonstrate improved communication and computational efficiency without sacrificing accuracy or trustworthiness, outperforming several baseline DFL methods under imperfect communication conditions. This approach offers a scalable, privacy-preserving pathway for robust DFL in complex, bandwidth-constrained networks.

Abstract

Decentralized Federated Learning (DFL) enables collaborative model training without a central server but faces challenges in efficiency, stability, and trustworthiness due to communication and computational limitations among distributed nodes. To address these critical issues, we introduce a sparsity constraint on the shared model, leading to Sparse DFL (SDFL), and propose a novel algorithm, CEPS. The sparsity constraint facilitates the use of one-bit compressive sensing to transmit one-bit information between partially selected neighbour nodes at specific steps, thereby significantly improving communication efficiency. Moreover, we integrate differential privacy into the algorithm to ensure privacy preservation and bolster the trustworthiness of the learning process. Furthermore, CEPS is underpinned by theoretical guarantees regarding both convergence and privacy. Numerical experiments validate the effectiveness of the proposed algorithm in improving communication and computation efficiency while maintaining a high level of trustworthiness.

Sparse Decentralized Federated Learning

TL;DR

This work tackles the efficiency, stability, and privacy challenges of decentralized federated learning (DFL) by introducing Sparse DFL (SDFL) and the CEPS algorithm. CEPS combines a sparsity constraint with 1-bit compressive sensing (1BCS) to drastically reduce communication, while employing inexact ADMM to enable efficient, distributed optimization with partial neighbor participation. Privacy is guaranteed via the Gaussian mechanism, providing -DP per iteration and cumulatively through adaptive composition, along with theoretical convergence guarantees under standard assumptions. Numerical experiments on linear and logistic regression tasks demonstrate improved communication and computational efficiency without sacrificing accuracy or trustworthiness, outperforming several baseline DFL methods under imperfect communication conditions. This approach offers a scalable, privacy-preserving pathway for robust DFL in complex, bandwidth-constrained networks.

Abstract

Decentralized Federated Learning (DFL) enables collaborative model training without a central server but faces challenges in efficiency, stability, and trustworthiness due to communication and computational limitations among distributed nodes. To address these critical issues, we introduce a sparsity constraint on the shared model, leading to Sparse DFL (SDFL), and propose a novel algorithm, CEPS. The sparsity constraint facilitates the use of one-bit compressive sensing to transmit one-bit information between partially selected neighbour nodes at specific steps, thereby significantly improving communication efficiency. Moreover, we integrate differential privacy into the algorithm to ensure privacy preservation and bolster the trustworthiness of the learning process. Furthermore, CEPS is underpinned by theoretical guarantees regarding both convergence and privacy. Numerical experiments validate the effectiveness of the proposed algorithm in improving communication and computation efficiency while maintaining a high level of trustworthiness.
Paper Structure (34 sections, 7 theorems, 71 equations, 7 figures, 5 tables, 1 algorithm)

This paper contains 34 sections, 7 theorems, 71 equations, 7 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Prior Arts
  3. Foundational DFL Algorithms
  4. Communication Efficiency
  5. Communication Topology
  6. Privacy Preservation
  7. Our contribution
  8. Organization
  9. Preliminaries
  10. Notation
  11. Decentralized Federated Learning
  12. Sparse DFL
  13. Differential Privacy
  14. DFL via Inexact ADM
  15. Algorithm Design
  16. ...and 19 more sections

Key Result

Theorem 1

Under Assumptions assump-gradientlipassump-varianceboundassump-1bcsassump-communicate and setting ${c\geq c_0}$, the following bound holds for Algorithm algorithm-CEPS, where $\varrho :=\sum_{i=1}^m\varrho_i$ and $e_{\infty}$ is an error bound relied on the sparse projection of each iteration and the added noise.

Figures (7)

  • Figure 1: Self-comparison of CEPS: Effect of $m$.
  • Figure 2: Self-comparison of CEPS: Effect of $\varepsilon$.
  • Figure 3: Self-comparison of CEPS: Effect of $r$.
  • Figure 4: Objective v.s. CR.
  • Figure 5: CR v.s. $m$.
  • ...and 2 more figures

Theorems & Definitions (21)

  • Definition 1
  • Definition 2
  • Theorem 1
  • proof
  • Theorem 2
  • proof
  • Theorem 3
  • proof
  • Example 1
  • Example 2
  • ...and 11 more