Unbiased Compression Saves Communication in Distributed Optimization: When and How Much?

TL;DR

This work analyzes how communication compression affects total communication cost in distributed optimization. It shows that unbiased compression alone cannot reduce total cost, due to extra rounds needed to compensate for distortion, but that independence across workers' compressors enables error cancellation and substantial savings, up to $\Theta\left(\sqrt{\min\{n,\kappa\}}\right)$. The authors formalize a total communication cost framework, derive tight lower bounds for independent unbiased compressors, and refine the convergence analysis of ADIANA to nearly match these bounds. Empirical results on synthetic data, logistic regression, and CIFAR-10 corroborate the theory, highlighting the practical value of independent unbiased compression. Collectively, the results guide when and how much compression helps and introduce ADIANA as a near-optimal algorithm under independence, advancing understanding of communication-efficiency in distributed optimization.

Abstract

Communication compression is a common technique in distributed optimization that can alleviate communication overhead by transmitting compressed gradients and model parameters. However, compression can introduce information distortion, which slows down convergence and incurs more communication rounds to achieve desired solutions. Given the trade-off between lower per-round communication costs and additional rounds of communication, it is unclear whether communication compression reduces the total communication cost. This paper explores the conditions under which unbiased compression, a widely used form of compression, can reduce the total communication cost, as well as the extent to which it can do so. To this end, we present the first theoretical formulation for characterizing the total communication cost in distributed optimization with communication compression. We demonstrate that unbiased compression alone does not necessarily save the total communication cost, but this outcome can be achieved if the compressors used by all workers are further assumed independent. We establish lower bounds on the communication rounds required by algorithms using independent unbiased compressors to minimize smooth convex functions and show that these lower bounds are tight by refining the analysis for ADIANA. Our results reveal that using independent unbiased compression can reduce the total communication cost by a factor of up to $Θ(\sqrt{\min\{n, κ\}})$ when all local smoothness constants are constrained by a common upper bound, where $n$ is the number of workers and $κ$ is the condition number of the functions being minimized. These theoretical findings are supported by experimental results.

Figures (6)

• Figure 1: Performance of ADIANA using random-$s$ sparsification compressors with shared (s.d.) or independent (i.d.) randomness against distributed Nesterov's accelerated algorithm with no compression in communication. Experimental descriptions are in Appendix \ref{['app:expri']}
• Figure 2: An illustration of the compressed aggregation.
• Figure 3: Convergence results of various distributed algorithms on a synthetic least squares problem (left), logistic regression problems with dataset a9a (middle) and w8a (right). The $y$-axis represents $f(\hat{x})-f^\star$ and the $x$-axis indicates the total communicated bits sent by per worker.
• Figure 4: Convergence results of various distributed algorithms on a synthetic least squares problem (left), logistic regression problems with dataset a9a (middle) and w8a (right). The $y$-axis represents $f(\hat{x})-f^\star$ and the $x$-axis indicates the total communicated bits sent by per worker. All compressors used are independent natural compression.
• Figure 5: Convergence results of various distributed algorithms on a synthetic least squares problem (left), logistic regression problems with dataset a9a (middle) and w8a (right). The $y$-axis represents $f(\hat{x})-f^\star$ and the $x$-axis indicates the total communicated bits sent by per worker. All compressors used are independent random quantization.

Theorems & Definitions (23)

• Definition 1: Algorithm class
• Remark 1
• Remark 2
• Proposition 1
• Lemma 1: he2023lower, Theorem 1, Informal
• Theorem 1
• Theorem 2
• Theorem 3
• Example 1: Random-$s$ sparsification
• Lemma 2: Safaryan2020UncertaintyPF, Theorem 2