Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Decentralized Federated Learning by Partial Message Exchange

Shan Sha, Shenglong Zhou, Xin Wang, Lingchen Kong, Geoffrey Ye Li

TL;DR

A novel algorithm, PaME (DFL by Partial Message Exchange), to allow only randomly selected sparse coordinates to be exchanged between two neighbor nodes, which achieves substantial reductions in communication costs while still preserving a high level of privacy, without sacrificing accuracy.

Abstract

Decentralized federated learning (DFL) has emerged as a transformative server-free paradigm that enables collaborative learning over large-scale heterogeneous networks. However, it continues to face fundamental challenges, including data heterogeneity, restrictive assumptions for theoretical analysis, and degraded convergence when standard communication- or privacyenhancing techniques are applied. To overcome these drawbacks, this paper develops a novel algorithm, PaME (DFL by Partial Message Exchange). The central principle is to allow only randomly selected sparse coordinates to be exchanged between two neighbor nodes. Consequently, PaME achieves substantial reductions in communication costs while still preserving a high level of privacy, without sacrificing accuracy. Moreover, grounded in rigorous analysis, the algorithm is shown to converge at a linear rate under the gradient to be locally Lipschitz continuous and the communication matrix to be doubly stochastic. These two mild assumptions not only dispense with many restrictive conditions commonly imposed by existing DFL methods but also enables PaME to effectively address data heterogeneity. Furthermore, comprehensive numerical experiments demonstrate its superior performance compared with several representative decentralized learning algorithms.

Decentralized Federated Learning by Partial Message Exchange

TL;DR

A novel algorithm, PaME (DFL by Partial Message Exchange), to allow only randomly selected sparse coordinates to be exchanged between two neighbor nodes, which achieves substantial reductions in communication costs while still preserving a high level of privacy, without sacrificing accuracy.

Abstract

Decentralized federated learning (DFL) has emerged as a transformative server-free paradigm that enables collaborative learning over large-scale heterogeneous networks. However, it continues to face fundamental challenges, including data heterogeneity, restrictive assumptions for theoretical analysis, and degraded convergence when standard communication- or privacyenhancing techniques are applied. To overcome these drawbacks, this paper develops a novel algorithm, PaME (DFL by Partial Message Exchange). The central principle is to allow only randomly selected sparse coordinates to be exchanged between two neighbor nodes. Consequently, PaME achieves substantial reductions in communication costs while still preserving a high level of privacy, without sacrificing accuracy. Moreover, grounded in rigorous analysis, the algorithm is shown to converge at a linear rate under the gradient to be locally Lipschitz continuous and the communication matrix to be doubly stochastic. These two mild assumptions not only dispense with many restrictive conditions commonly imposed by existing DFL methods but also enables PaME to effectively address data heterogeneity. Furthermore, comprehensive numerical experiments demonstrate its superior performance compared with several representative decentralized learning algorithms.
Paper Structure (34 sections, 16 theorems, 163 equations, 12 figures, 2 tables, 4 algorithms)

This paper contains 34 sections, 16 theorems, 163 equations, 12 figures, 2 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Neighbor Selection
  4. Parameter Aggregation
  5. Local Update
  6. Contributions
  7. Organization
  8. Preliminaries
  9. Notation
  10. Decentralized Federated Learning
  11. DFL via Inexact ADM
  12. Algorithm Design
  13. Partial Message Exchange (PME)
  14. Communication Efficiency
  15. Privacy Enhancement
  16. ...and 19 more sections

Key Result

Theorem 1

Let $\overline{\mathbf{w}}$ be the average of $q$ vectors $\mathbf{w}_1$, $\mathbf{w}_2$, $\cdots$, ${\mathbf{w}_q \in \mathbb{R}^n}$. For each ${i\in[q]}$, independently construct a sparse vector ${\mathbf{v}_i\in\mathbb{R}^n}$ by uniformly selecting $s$ coordinates of $\mathbf{w}_i$ without replac and two averages $\overline{\mathbf{v}}$ and $\widetilde{\mathbf{v}}$ by for any $\ell\in[n]$. The

Figures (12)

  • Figure 1: Partial message exchange (PME) : Every neighbor of a local node $i$ randomly selects partial coordinates (or messages) and transmits them to node $i$. Node $i$ averages the received incomplete parameters and fills in the missing coordinates using the coordinates in its own local parameter.
  • Figure 2: Self-comparison of PaME.
  • Figure 3: Self comparison of PaME: Convergence curve of different transmission rate.
  • Figure 4: Self comparison of PaME: Convergence curve of different participation rate.
  • Figure 5: Self comparison of PaME: Convergence curve of different CPs (homogeneous).
  • ...and 7 more figures

Theorems & Definitions (34)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Example 1
  • Example 2
  • Example 3
  • Example 4
  • Theorem 1
  • proof
  • ...and 24 more