Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BiCoLoR: Communication-Efficient Optimization with Bidirectional Compression and Local Training

Laurent Condat, Artavazd Maranjyan, Peter Richtárik

TL;DR

BiCoLoR tackles the communication bottleneck in distributed optimization by marrying Local Training with bidirectional unbiased compression. It introduces a stochastic primal–dual framework that decouples uplink and downlink compression and uses dual variables for variance reduction, achieving accelerated $TotalCom$ complexity in both strongly convex and general convex heterogeneous settings. Theoretical results show linear convergence when $\mu>0$ and accelerated sublinear convergence in the general convex case, with $TotalCom$ scaling $\tilde{\mathcal{O}}(d\sqrt{\kappa})$ under practical compression, and with decoupled uplink/downlink variances. Empirically, BiCoLoR outperforms existing bidirectional-CC baselines on logistic regression benchmarks, setting a new standard for communication-efficient distributed optimization.

Abstract

Slow and costly communication is often the main bottleneck in distributed optimization, especially in federated learning where it occurs over wireless networks. We introduce BiCoLoR, a communication-efficient optimization algorithm that combines two widely used and effective strategies: local training, which increases computation between communication rounds, and compression, which encodes high-dimensional vectors into short bitstreams. While these mechanisms have been combined before, compression has typically been applied only to uplink (client-to-server) communication, leaving the downlink (server-to-client) side unaddressed. In practice, however, both directions are costly. We propose BiCoLoR, the first algorithm to combine local training with bidirectional compression using arbitrary unbiased compressors. This joint design achieves accelerated complexity guarantees in both convex and strongly convex heterogeneous settings. Empirically, BiCoLoR outperforms existing algorithms and establishes a new standard in communication efficiency.

BiCoLoR: Communication-Efficient Optimization with Bidirectional Compression and Local Training

TL;DR

BiCoLoR tackles the communication bottleneck in distributed optimization by marrying Local Training with bidirectional unbiased compression. It introduces a stochastic primal–dual framework that decouples uplink and downlink compression and uses dual variables for variance reduction, achieving accelerated complexity in both strongly convex and general convex heterogeneous settings. Theoretical results show linear convergence when and accelerated sublinear convergence in the general convex case, with scaling under practical compression, and with decoupled uplink/downlink variances. Empirically, BiCoLoR outperforms existing bidirectional-CC baselines on logistic regression benchmarks, setting a new standard for communication-efficient distributed optimization.

Abstract

Slow and costly communication is often the main bottleneck in distributed optimization, especially in federated learning where it occurs over wireless networks. We introduce BiCoLoR, a communication-efficient optimization algorithm that combines two widely used and effective strategies: local training, which increases computation between communication rounds, and compression, which encodes high-dimensional vectors into short bitstreams. While these mechanisms have been combined before, compression has typically been applied only to uplink (client-to-server) communication, leaving the downlink (server-to-client) side unaddressed. In practice, however, both directions are costly. We propose BiCoLoR, the first algorithm to combine local training with bidirectional compression using arbitrary unbiased compressors. This joint design achieves accelerated complexity guarantees in both convex and strongly convex heterogeneous settings. Empirically, BiCoLoR outperforms existing algorithms and establishes a new standard in communication efficiency.
Paper Structure (18 sections, 4 theorems, 64 equations, 3 figures, 1 table, 1 algorithm)

This paper contains 18 sections, 4 theorems, 64 equations, 3 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Unbiased compression
  3. Problem formulation
  4. Challenge and contribution
  5. Related work
  6. Local training
  7. Compression
  8. Combining LT and CC
  9. Proposed algorithm 0.9BiCoLoR
  10. Convergence analysis and complexity of 0.9BiCoLoR
  11. Accelerated linear convergence in the strongly convex case
  12. Accelerated sublinear convergence in the general convex case
  13. Discussion of the $\tilde{\mathcal{O}}(d\sqrt{\kappa})$ TotalCom complexity
  14. Experiments
  15. Strongly convex case
  16. ...and 3 more sections

Key Result

Theorem 4.1

Suppose that $\mu>0$ and let $x^\star$ be the unique solution to eq1. In 0.9BiCoLoR, suppose that Assumption ass2 holds, $0<\gamma < \frac{2}{L}$, $p_t\equiv p\in (0,1]$ is constant, and For every $t\geq 0$, define the Lyapunov function where $u_y^\star \coloneqq \nabla g(x^\star)$ and $u_i^\star \coloneqq \nabla f_i(x^\star)$. Then 0.9BiCoLoR converges linearly: for every $t\geq 0$, In additio

Figures (3)

  • Figure 1: Logistic regression on the real-sim dataset. The compression scheme combines sparsification with $K = 1000$ and Natural Compression.
  • Figure 2: Logistic regression on the w8a dataset. The compression scheme combines sparsification with $K = 100$ and Natural Compression.
  • Figure 3: Logistic regression without regularization on the w8a dataset. The compression scheme combines sparsification with $K = 100$ and Natural Compression. 0.9BiCoLoR with decreasing $p_t$ (as defined in Theorem \ref{['theogc']}) outperforms the constant-$p$ variant in the long run.

Theorems & Definitions (4)

  • Theorem 4.1: linear convergence of 0.9BiCoLoR
  • Corollary 4.2
  • Corollary 4.3
  • Theorem 4.4: accelerated sublinear convergence of 0.9BiCoLoR