Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Controlled disagreement improves generalization in decentralized training

Zesen Wang, Mikael Johansson

TL;DR

This work introduces decentralized SGD with Adaptive Consensus (DSGD-AC), which intentionally preserves non-vanishing consensus errors through a time-dependent scaling mechanism, and establishes consensus errors as a useful implicit regularizer and opens a new perspective on the design of decentralized learning algorithms.

Abstract

Decentralized training is often regarded as inferior to centralized training because the consensus errors between workers are thought to undermine convergence and generalization, even with homogeneous data distributions. This work challenges this view by introducing decentralized SGD with Adaptive Consensus (DSGD-AC), which intentionally preserves non-vanishing consensus errors through a time-dependent scaling mechanism. We prove that these errors are not random noise but systematically align with the dominant Hessian subspace, acting as structured perturbations that guide optimization toward flatter minima. Across image classification and machine translation benchmarks, DSGD-AC consistently surpasses both standard DSGD and centralized SGD in test accuracy and solution flatness. Together, these results establish consensus errors as a useful implicit regularizer and open a new perspective on the design of decentralized learning algorithms.

Controlled disagreement improves generalization in decentralized training

TL;DR

This work introduces decentralized SGD with Adaptive Consensus (DSGD-AC), which intentionally preserves non-vanishing consensus errors through a time-dependent scaling mechanism, and establishes consensus errors as a useful implicit regularizer and opens a new perspective on the design of decentralized learning algorithms.

Abstract

Decentralized training is often regarded as inferior to centralized training because the consensus errors between workers are thought to undermine convergence and generalization, even with homogeneous data distributions. This work challenges this view by introducing decentralized SGD with Adaptive Consensus (DSGD-AC), which intentionally preserves non-vanishing consensus errors through a time-dependent scaling mechanism. We prove that these errors are not random noise but systematically align with the dominant Hessian subspace, acting as structured perturbations that guide optimization toward flatter minima. Across image classification and machine translation benchmarks, DSGD-AC consistently surpasses both standard DSGD and centralized SGD in test accuracy and solution flatness. Together, these results establish consensus errors as a useful implicit regularizer and open a new perspective on the design of decentralized learning algorithms.
Paper Structure (40 sections, 5 theorems, 78 equations, 17 figures, 8 tables)

This paper contains 40 sections, 5 theorems, 78 equations, 17 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Canonical view about consensus errors
  4. Explorations beyond the canonical view
  5. Explicit curvature-related perturbations improve generalization but are costly
  6. Problem setting and notation
  7. Decentralized optimization.
  8. Decentralized SGD (DSGD)
  9. DSGD-AC: DSGD with adaptive consensus
  10. Finding: vanishing consensus error in DSGD
  11. Algorithm: DSGD with adaptive consensus
  12. Controlled consensus errors in DSGD-AC
  13. Consensus errors align with dominant subspace of Hessian
  14. Numerical Experiments
  15. Ablation study and sensitivity analysis
  16. ...and 25 more sections

Key Result

Proposition 3.1

In a quasi-stationary regime with mild bounded-moment and spectral assumptions (see Appendix app:radius) the disagreement radius satisfies In particular, if $\alpha^{(t)}\rightarrow 0$ and $\gamma^{(t)}$ is bounded away from zero, then $r_t^2\rightarrow 0$. Thus, no constant $\gamma^{(t)}$ can maintain a non-vanishing disagreement radius as the step size diminishes. However, if we choose $\gamma^

Figures (17)

  • Figure 1: Decentralized training of WRN28-10 on CIFAR-10 with 8 workers and the one-peer ring topology. $p=3$ for DSGD-AC.
  • Figure 2: Decentralized SGD with adaptive consensus (DSGD-AC) on worker $i$
  • Figure 3: Losses on the whole training dataset at local workers and global average. The losses are evaluated every 10 epochs.
  • Figure 4: Training loss at epoch 180 along: (1) worker i: lines connecting global average and worker $i$, (2) gradient: the line that aligns with the full-batch gradient at the global average and crosses the global average, and (3) random: 500 lines that cross the global average and follow random directions generated as in bisla2022low. The $x$-axis means the directional magnitude of the perturbation along these directions. The red dots represent the losses at the local models. The losses are computed on $\sim1/4$ of the training dataset due to heavy computation complexity.
  • Figure 5: Average norm of consensus errors over epochs with varying $p$ in the WRN28-10 on CIFAR-10 experiments with 8 workers and the one-peer ring topology.
  • ...and 12 more figures

Theorems & Definitions (5)

  • Proposition 3.1: Disagreement radius and the role of $\gamma$
  • Proposition 3.2: Structure of consensus errors
  • Lemma 1.1: Local loss envelope
  • Proposition 1.2: Curvature tilt
  • Corollary 1.3: Hessian-aligned mini-batch noise