On the Surprising Effectiveness of a Single Global Merging in Decentralized Learning

TL;DR

The paper investigates temporal scheduling of communication in fully decentralized learning with non-IID data, revealing that concentrating communication toward the end of training and performing a single final global merge can drastically improve global generalization, approaching the performance of parallel SGD under severe communication constraints. A theoretical framework shows that the globally merged DSGD model can match parallel SGD’s convergence rate by reinterpretating inter-agent discrepancies as constructive bias, and it introduces a sufficient condition, the Critical Consensus Edge, guiding when to increase communication. Empirically, mergeability persists under limited nonzero communication across diverse datasets, models, and topologies, offering a principled basis for adaptive, communication-efficient decentralized training and model merging research. The work suggests practical strategies for scalable distributed learning and connects decentralized optimization dynamics to landscape geometry and basin connectivity, with reproducible experiments and open-source code forthcoming.

Abstract

Decentralized learning provides a scalable alternative to parameter-server-based training, yet its performance is often hindered by limited peer-to-peer communication. In this paper, we study how communication should be scheduled over time to improve global generalization, including determining when and how frequently devices synchronize. Counterintuitive empirical results show that concentrating communication budgets in the later stages of decentralized training remarkably improves global generalization. Surprisingly, we uncover that fully connected communication at the final step, implemented by a single global merging, can significant improve the generalization performance of decentralized learning under serve high data heterogeneity. Our theoretical contributions, which explains these phenomena, are first to establish that the globally merged model of decentralized SGD can match the convergence rate of parallel SGD. Technically, we reinterpret part of the discrepancy among local models, which were previously considered as detrimental noise, as constructive components essential for matching this rate. This work provides promising results that decentralized learning is able to generalize under high data heterogeneity and limited communication, while offering broad new avenues for model merging research. The code will be made publicly available.

Figures (7)

• Figure 1: (a, b): Global test accuracy (see \ref{['def: avg_gen']}) of CLIP ViT-B/32 (a) and ResNet-18 (b) trained on Tiny ImageNet using FedAvg (blue), decentralized SGD (orange), and one-shot FedAvg (green), distributed across 32 agents with high data heterogeneity (Dirichlet $\alpha=0.1$). Decentralized training involves each agent syncing model parameters with a random peer per round with a probability of 0.2, with a single global merging at the final round (see details in \ref{['sec: setup']}). (c) Loss landscape for 16-agent decentralized training, prior to the final merging (see \ref{['sec: setup']} for visualization details).(d): An illustration comparing federated, decentralized, and local training.
• Figure 2: (a, b): Comparisons of global test accuracy (see \ref{['def: avg_gen']}) in decentralized training of ResNet-18 on CIFAR-100 with AdamW, distributed across 16 agents with Dirichlet $\alpha$ = 0.1 (see details in \ref{['sec: setup']}). Fully-connected communication (i.e., AllReduce) is activated only in specific windows, while low communication with one random peer with a probability of $0.2$ is used elsewhere. (a): Fully-connected communication in $1/10$ of total rounds. (b): Fully-connected communication in $1/20$ of total rounds. In both, lighter bars show peak accuracy, darker bars show final accuracy. (c): Global test accuracy curves for local models and the globally averaged model (counterfactual) under persistent low communication (blue) and no communication (orange).
• Figure C.1: (a, b): Global test accuracy (see \ref{['def: avg_gen']}) of CLIP ViT-B/32 (a) and ResNet-18 (b) trained on Tiny ImageNet using FedAdamW (blue), decentralized AdamW (orange), and one-shot FedAdamW (green), distributed across 16 agents with high data heterogeneity (Dirichlet $\alpha=0.1$). Decentralized training involves each agent syncing model parameters with a random peer per round with a probability of 0.2, with a single global merging at the final round (see details in \ref{['sec: setup']}).
• Figure C.2: (a, b): Global test accuracy (see \ref{['def: avg_gen']}) of CLIP ViT-B/32 (a) and ResNet-18 (b) trained on Tiny ImageNet using FedAdamW (blue), decentralized AdamW (orange), and one-shot FedAdamW (green), distributed across 32 agents with high data heterogeneity (Dirichlet $\alpha=0.1$). Decentralized training involves each agent syncing model parameters with a random peer per round with a probability of 0.2, with a single global merging at the final round (see details in \ref{['sec: setup']}).
• Figure C.3: Global test accuracy (see \ref{['def: avg_gen']}) of training ResNet-18 on Tiny ImageNet, distributed across 16 agents with high heterogeneity (Dirichlet $\alpha$ = 0.1; see details in \ref{['sec: setup']}). We evaluate the effects of different (a) number of peers $R$, and (b) communication topologies. Pretrained weights are used only in (a).

Theorems & Definitions (30)

• Definition 1: Average Global Test Accuracy
• Remark 1: Metric Justification
• Definition 2: Mergeability under Global Population Risk
• Theorem 1: Non-convex Convergence Rate of DSGD
• Remark 2
• Remark 3
• Remark 4
• Proposition 1
• Remark 5
• Proposition 2: Critical Consensus Edge