Markovian Compression: Looking to the Past Helps Accelerate the Future

TL;DR

This work tackles the communication bottleneck in distributed optimization by introducing history-aware Markovian compressors, specifically BanLast and Kawasaki, that depend on past iterations.They integrate these into QSGD (MQSGD) and a momentum-accelerated variant (AMQSGD) and establish convergence guarantees across nonconvex, PL, and strongly convex regimes, with bounds that explicitly involve the Markov chain mixing time. Empirical results on CIFAR-10 and GLUE (ResNet-18, DeBERTaV3-base) demonstrate faster convergence and improved performance over unbiased and competing compression schemes, validating practical effectiveness despite theoretical looseness relative to traditional unbiased compression. The findings highlight a viable path to faster, scalable distributed optimization under communication constraints, with clear guidance on how history length and forgetting factors influence performance. Overall, the proposed Markovian compression framework broadens the toolkit for distributed learning by leveraging information from previous transmissions to accelerate future iterations.

Abstract

This paper deals with distributed optimization problems that use compressed communication to achieve efficient performance and mitigate communication bottleneck. We propose a family of compression schemes in which operators transform vectors fed to their input according to a Markov chain, i.e. the stochasticity of the compressors depends on previous iterations. The compressors are implemented in the vanilla Quantized Stochastic Gradient Descent algorithm (QSGD), and, to further improve the efficiency and convergence rate, in the momentum accelerated QSGD. We provide convergence results for our algorithms with Markovian compressors, the analysis covers non-convex, Polyak-Lojasiewicz, and strongly convex cases. To demonstrate the applicability of our approach to distributed data-parallel optimization problems, we conduct experiments on the CIFAR-10 and GLUE datasets with the Resnet-18 and DeBERTaV3 models. Practical results show the superiority of methods that use our compressor design over existing schemes.

Figures (16)

• Figure 1: Logistic Regression on MNIST experiments results. Best runs for each method are displayed.
• Figure 2: Image classification with ResNet-18 on CIFAR-10 experiments results. Best runs for each method are displayed.
• Figure 3: Comparison with other compressors on Resnet-18 training on CIFAR-10 dataset for Rand5% sparsification on $n=20$ clients. Natural compression factor is 4. Left figure is sequential combination with Natural compression. Right figure is comparison against PermK and Natural compression independently.
• Figure 4: Comparison of the performance of BanLast, KAWASAKI and Randm on the fine-tuning task on a subset of GLUE benchmark for $n=10$ clients.
• Figure 5: Theoretical estimate on dependence of history buffer size $K$ on parameter $\alpha = d/m$: (a) represents expected number of iterations required to transfer all coordinates to server on history buffer size $K$ for different $\alpha$, (b) represents scaling of optimal history buffer size $K^*$ on $\alpha$. (c) represents comparison of expected number of iterations required to transfer all coordinates to server on problems parameter $\alpha$ for Rand$m$ and BanLast$K$.

Theorems & Definitions (41)

• definition 1: Markov chain
• definition 2: Ergodicity of Markov chain
• definition 3: Mixing time of the discrete Markov chain
• definition 4: Random sparsification
• definition 5: BanLast($K, m$) compressor
• definition 6: KAWASAKI($K, b, \pi_\Delta, m$) compressor
• Example 1
• theorem 2: Asymptotic unbiasedness of BanLast($K, m$) and KAWASAKI($K, b, \pi_\Delta, m$)
• theorem 3: Convergence of MQSGD (Algorithm \ref{['alg:GD']})
• Corollary 4: Step tuning for Theorem \ref{['theorem:GD_odd']}