Local Steps Speed Up Local GD for Heterogeneous Distributed Logistic Regression

TL;DR

This work investigates whether local updates can speed up distributed optimization for heterogeneous logistic regression. It introduces a two-stage Local GD with learning-rate warmup that achieves a last-iterate convergence rate of $F(\hat{\boldsymbol{w}}_2)=\tilde{O}(1/(\gamma^2 K R))$ after a warmup of $\tilde{O}(KM/\gamma^4)$ rounds, and analyzes a Local Gradient Flow variant to establish progress under fixed stepsizes. The theoretical results reveal that acceleration via local steps can arise from the loss landscape itself, not merely objective similarity, and the warmup mechanism is key to stability when using large local steps. Complementary experiments on synthetic data, MNIST, and CIFAR-10 support the practical benefits of local updates and illustrate the nuanced role of data heterogeneity and step-size strategies. Overall, the paper provides a refined, problem-specific view of Local SGD that can inform both theory and practice in distributed optimization.

Abstract

We analyze two variants of Local Gradient Descent applied to distributed logistic regression with heterogeneous, separable data and show convergence at the rate $O(1/KR)$ for $K$ local steps and sufficiently large $R$ communication rounds. In contrast, all existing convergence guarantees for Local GD applied to any problem are at least $Ω(1/R)$, meaning they fail to show the benefit of local updates. The key to our improved guarantee is showing progress on the logistic regression objective when using a large stepsize $η\gg 1/K$, whereas prior analysis depends on $η\leq 1/K$.

Figures (2)

• Figure 1: Train loss of Local GD for a synthetic dataset and MNIST. Left: Small stepsize $\eta = 1/(KH)$, as required by baselines (Corollary \ref{['cor:generic_local_sgd_rate_local']}). Middle: Two stage stepsize with $\eta_1 = 1/(KH)$ and $\eta_2 = 1/H$, as in our Theorem \ref{['thm:two_stage_convergence']}. Right: Large stepsize $\eta = 1/H$. For the synthetic dataset, a large stepsize causes the loss to increase significantly during early rounds.
• Figure 2: Train loss and testing accuracy for heterogeneous, distributed CIFAR-10 with ResNet-50.

Theorems & Definitions (80)

• Corollary 1
• Corollary 2
• Theorem 1
• Corollary 3
• Lemma 1
• Lemma 2
• Lemma 3
• Lemma 4
• Theorem 2
• Lemma 5