LASER: Linear Compression in Wireless Distributed Optimization

TL;DR

LASER tackles communication bottlenecks in wireless distributed optimization by exploiting low-rank gradient structure for linear compression and noisy-channel transmission. It uses PowerSGD to produce rank-$r$ factors, performs error feedback, and allocates power across factors to improve signal fidelity; the server reconstructs the gradient as a product of received factors. Theoretical analysis shows convergence rates matching SGD up to a channel-dependent additive term $\lambda_{\text{LASER}}$, significantly smaller than $\lambda_{\text{Z-SGD}}$ under typical regimes. Empirically, LASER yields substantial perplexity and accuracy gains on GPT language modeling and image classification under noise, while reducing data transfer by up to two orders of magnitude, demonstrating practical viability for wireless distributed learning.

Abstract

Data-parallel SGD is the de facto algorithm for distributed optimization, especially for large scale machine learning. Despite its merits, communication bottleneck is one of its persistent issues. Most compression schemes to alleviate this either assume noiseless communication links, or fail to achieve good performance on practical tasks. In this paper, we close this gap and introduce LASER: LineAr CompreSsion in WirEless DistRibuted Optimization. LASER capitalizes on the inherent low-rank structure of gradients and transmits them efficiently over the noisy channels. Whilst enjoying theoretical guarantees similar to those of the classical SGD, LASER shows consistent gains over baselines on a variety of practical benchmarks. In particular, it outperforms the state-of-the-art compression schemes on challenging computer vision and GPT language modeling tasks. On the latter, we obtain $50$-$64 \%$ improvement in perplexity over our baselines for noisy channels.

Figures (7)

• Figure 1: Final test perplexity after 20k iterations (lower is better) vs. power budget for GPT-2 language modeling on WikiText-103. LASER consistently requires orders-of-magnitude less power than other methods for the same perplexity.
• Figure 2: Test accuracy (higher the better) for a given power budget on Cifar-10 for different algorithms. LASER demonstrates consistent accuracy gains over the baselines over a wide range of power levels.
• Figure 3: Accuracy vs. budget $P$ for various laws. Constant is the best for both LASER and Z-SGD.
• Figure 4: Test accuracy (higher the better) for a given power budget on Cifar-100 for different algorithms. The advantage of LASER is evident across the entire power spectrum.
• Figure 5: Final accuracy vs. compression rank tradeoff for CIFAR-10 classification, for low, medium and high power regimes. Rank-$4$/Rank-$8$ compression is optimal for all the three regimes. It reveals two interesting insights: (i) performance is uniformly worse in all the regimes with overly aggressive rank-one compression, and (ii) higher rank compression impacts low power regime more significantly than the medium and high-power counterparts. This confirms with the intuition that at low power (and hence noisier channel), it is better to allocate the limited power budget appropriately to few "essential" rank components as opposed to thinning it out over many.

Theorems & Definitions (28)

• Definition 1: Channel influence factor
• Theorem 1: LASER convergence
• proof
• Lemma 1: stich2020error, Lemma 8
• Lemma 2: stich2020error, Lemma 22
• Lemma 3: stich2020error, Lemma 13
• Theorem 2: stich2020error, Theorem 22
• Lemma 4: stich2020error, Lemma 9
• Lemma 5: stich2020error, Lemma 22
• Lemma 6: stich2020error, Lemma 14