A Unified Convergence Analysis for Semi-Decentralized Learning: Sampled-to-Sampled vs. Sampled-to-All Communication

TL;DR

The paper tackles semi-decentralized federated learning with two server-to-device dissemination strategies, Sampled-to-Sampled (S2S) and Sampled-to-All (S2A). It develops a unified convergence framework that captures intra- and inter-component heterogeneity, sampling rate, server period, and network connectivity, deriving bias/disagreement decompositions and convex/non-convex bounds. Theoretical results identify regimes where S2S or S2A dominates, and extensive MNIST/CIFAR experiments validate practical deployment guidelines. The findings offer concrete guidance for configuring semi-decentralized FL to balance bias and disagreement under realistic network constraints.

Abstract

In semi-decentralized federated learning, devices primarily rely on device-to-device communication but occasionally interact with a central server. Periodically, a sampled subset of devices uploads their local models to the server, which computes an aggregate model. The server can then either (i) share this aggregate model only with the sampled clients (sampled-to-sampled, S2S) or (ii) broadcast it to all clients (sampled-to-all, S2A). Despite their practical significance, a rigorous theoretical and empirical comparison of these two strategies remains absent. We address this gap by analyzing S2S and S2A within a unified convergence framework that accounts for key system parameters: sampling rate, server aggregation frequency, and network connectivity. Our results, both analytical and experimental, reveal distinct regimes where one strategy outperforms the other, depending primarily on the degree of data heterogeneity across devices. These insights lead to concrete design guidelines for practical semi-decentralized FL deployments.

Figures (15)

• Figure 1: Convergence rates for S2S and S2A, comparing Eqs. \ref{['eq:bound_sts_2']}--\ref{['eq:bound_sta_2']} with $n\!=\!100$, $L\!=\!f_0\!=\!1$, $\bar{\sigma}\!=\!0$. Left column: Sampling rate ($K/n$) with $H\!=\!5$, $p\!=\!1$. Center column: Server period ($H$) with $K/n\!=\!0.2$, $p\!=\!1$. Right column: Mixing parameter ($p$) with $K/n\!=\!0.2$, $H\!=\!5$.
• Figure 2: Test accuracy on MNIST dataset. Left column: Sampling rate ($K/n$) with $H=5$. Right column: Server period ($H$) with $K/n=0.2$.
• Figure 3: Test accuracy on CIFAR-10 dataset. Left column: Sampling rate ($K/n$) with $H=5$. Right column: Server period ($H$) with $K/n=0.2$.
• Figure 4: Accuracy gap on CIFAR-10 with ring topology.
• Figure 5: Test accuracy over communication rounds for intra IID, inter non-IID heterogeneity, $K/n\!=\!0.2$, ring topology.

Theorems & Definitions (32)

• Lemma 1: Orthogonal decomposition
• Lemma 2: Intra-component mixing parameter
• Theorem 1: Sampled-to-Sampled
• Theorem 2: Sampled-to-All
• Lemma 3: Descent Lemma (Convex Objectives)
• proof : Proof of Lemma \ref{['lem:descent']}
• Lemma 4: Descent Lemma (Non-Convex Objectives)
• proof : Proof of Lemma \ref{['lem:descent_nc']}
• Lemma 5: Disagreement Errors (D2D)
• proof : Proof of Lemma \ref{['consensus_intra_inter']}