Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LoCoDL: Communication-Efficient Distributed Learning with Local Training and Compression

Laurent Condat, Artavazd Maranjyan, Peter Richtárik

TL;DR

This work addresses communication bottlenecks in distributed optimization and Federated Learning by introducing 0.9LoCoDL, a randomized primal–dual method that combines Local Training and unidirectional Compression for uplink efficiency. By lifting the problem to a consensus formulation over $x_i$ and a shared $y$, and compressing only the differences to $y$, the method achieves linear convergence with a doubly-accelerated uplink complexity that scales favorably with the condition number $\kappa$ and model dimension $d$. The compressor class $\mathbb{U}(\omega)$ and the variance parameter $\omega_{\mathrm{av}}$ enable broad practical applicability with independent, unbiased quantizers, while the theoretical rate matches the best prior LT+CC results under mild assumptions. Empirically, 0.9LoCoDL shows superior communication efficiency across multiple datasets and compression schemes, outperforming existing LT+CC approaches and even some state-of-the-art accelerated methods, highlighting the value of combining Local Training with general unbiased compression in Federated/Distributed learning.

Abstract

In Distributed optimization and Learning, and even more in the modern framework of federated learning, communication, which is slow and costly, is critical. We introduce LoCoDL, a communication-efficient algorithm that leverages the two popular and effective techniques of Local training, which reduces the communication frequency, and Compression, in which short bitstreams are sent instead of full-dimensional vectors of floats. LoCoDL works with a large class of unbiased compressors that includes widely-used sparsification and quantization methods. LoCoDL provably benefits from local training and compression and enjoys a doubly-accelerated communication complexity, with respect to the condition number of the functions and the model dimension, in the general heterogenous regime with strongly convex functions. This is confirmed in practice, with LoCoDL outperforming existing algorithms.

LoCoDL: Communication-Efficient Distributed Learning with Local Training and Compression

TL;DR

This work addresses communication bottlenecks in distributed optimization and Federated Learning by introducing 0.9LoCoDL, a randomized primal–dual method that combines Local Training and unidirectional Compression for uplink efficiency. By lifting the problem to a consensus formulation over and a shared , and compressing only the differences to , the method achieves linear convergence with a doubly-accelerated uplink complexity that scales favorably with the condition number and model dimension . The compressor class and the variance parameter enable broad practical applicability with independent, unbiased quantizers, while the theoretical rate matches the best prior LT+CC results under mild assumptions. Empirically, 0.9LoCoDL shows superior communication efficiency across multiple datasets and compression schemes, outperforming existing LT+CC approaches and even some state-of-the-art accelerated methods, highlighting the value of combining Local Training with general unbiased compression in Federated/Distributed learning.

Abstract

In Distributed optimization and Learning, and even more in the modern framework of federated learning, communication, which is slow and costly, is critical. We introduce LoCoDL, a communication-efficient algorithm that leverages the two popular and effective techniques of Local training, which reduces the communication frequency, and Compression, in which short bitstreams are sent instead of full-dimensional vectors of floats. LoCoDL works with a large class of unbiased compressors that includes widely-used sparsification and quantization methods. LoCoDL provably benefits from local training and compression and enjoys a doubly-accelerated communication complexity, with respect to the condition number of the functions and the model dimension, in the general heterogenous regime with strongly convex functions. This is confirmed in practice, with LoCoDL outperforming existing algorithms.
Paper Structure (14 sections, 2 theorems, 40 equations, 7 figures, 2 tables, 1 algorithm)

This paper contains 14 sections, 2 theorems, 40 equations, 7 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem and Motivation
  3. State of the Art
  4. A General Class of Unbiased Random Compressors
  5. Challenge and Contributions
  6. Proposed Algorithm 0.9LoCoDL
  7. Principle: Double Lifting of the Problem to a Consensus Problem
  8. Description of 0.9LoCoDL
  9. Convergence and Complexity of 0.9LoCoDL
  10. The Case $g=0$
  11. Experiments
  12. Conclusion
  13. Proof of Theorem \ref{['theo1']}
  14. Additional Experiments

Key Result

Theorem 3.1

Suppose that Assumptions ass1 and ass2 hold. In 0.9LoCoDL, suppose that $0<\gamma < \frac{2}{L}$, $2\rho-\rho^2(1+\omega_{\mathrm{av}})-\chi\geq 0$. For every $t\geq 0$, define the Lyapunov function where $v^\star \coloneqq \nabla g(x^\star)$ and $u_i^\star \coloneqq \nabla f_i(x^\star)$. Then 0.9LoCoDL converges linearly: for every $t\geq 0$, In addition, for every $i\in[n]$, $(x_i^t)_{t\in\mat

Figures (7)

  • Figure 1: Comparison of several algorithms with several compressors on logistic regression with the 'a5a' dataset from the LibSVM, which has $d=122$ and 6,414 data points. We chose different values of $n$ to illustrate the two regimes $n<d$ and $n>d$, as discussed at the end of Section \ref{['seccc']}.
  • Figure 2: Comparison of several algorithms with several compressors on logistic regression with the 'diabetes' dataset from the LibSVM, which has $d=8$ and 768 data points. We chose different values of $n$ to illustrate the three regimes $n<d$, $n>d$, $n>d^2$, as discussed at the end of Section \ref{['seccc']}.
  • Figure 3: Comparison of several algorithms with various compressors on logistic regression with the 'w1a' dataset from the LibSVM, which has $d=300$ and 2,477 data points. We chose different values of $n$ to illustrate the two regimes, $n<d$ and $n>d$, as discussed at the end of \ref{['seccc']}.
  • Figure 4: Comparison of several algorithms with various compressors on logistic regression with the 'australian' dataset from the LibSVM, which has $d=14$ and 690 data points. We chose different values of $n$ to illustrate the three regimes: $n<d$, $n>d$, $n>d^2$, as discussed at the end of Section \ref{['seccc']}.
  • Figure 5: comparison of 0.9LoCoDL and 0.9ADIANA using several compressors for logistic regression with the 'covtype.binary' dataset from the LibSVM, which has $d=54$ and 581,010 data points. We chose different values of $n$ to illustrate the three regimes $n<d$, $n>d$, $n>d^2$, as discussed at the end of \ref{['seccc']}.
  • ...and 2 more figures

Theorems & Definitions (3)

  • Remark 2.2: partial participation
  • Theorem 3.1: linear convergence of 0.9LoCoDL
  • Corollary 3.2