Communication Compression for Distributed Learning with Aggregate and Server-Guided Feedback

TL;DR

This work tackles the uplink bottleneck in distributed and Federated Learning by proposing two error-feedback-free biased compression schemes, CAFe and CAFe-S, that rely on a globally shared predictor. CAFe uses the previous round's aggregated update as a predictor, while CAFe-S augments this with a server-side dataset to produce a potentially more accurate predictor. In non-convex settings with bounded gradient dissimilarity, CAFe achieves a convergence improvement by a factor of $(1-\omega)$ over standard compressed gradient descent, and CAFe-S attains a rate that improves as server-data become more representative, without imposing extra learning-rate penalties. Empirical results on MNIST, EMNIST, CIFAR-10, and CIFAR-100 show that CAFe consistently outperforms direct compression, with CAFe-S offering further gains when server and client data distributions align. Overall, the proposed frameworks enable efficient, privacy-preserving, large-scale Federated Learning with biased compression in practical edge settings.

Abstract

Distributed learning, particularly Federated Learning (FL), faces a significant bottleneck in the communication cost, particularly the uplink transmission of client-to-server updates, which is often constrained by asymmetric bandwidth limits at the edge. Biased compression techniques are effective in practice, but require error feedback mechanisms to provide theoretical guarantees and to ensure convergence when compression is aggressive. Standard error feedback, however, relies on client-specific control variates, which violates user privacy and is incompatible with stateless clients common in large-scale FL. This paper proposes two novel frameworks that enable biased compression without client-side state or control variates. The first, Compressed Aggregate Feedback (CAFe), uses the globally aggregated update from the previous round as a shared control variate for all clients. The second, Server-Guided Compressed Aggregate Feedback (CAFe-S), extends this idea to scenarios where the server possesses a small private dataset; it generates a server-guided candidate update to be used as a more accurate predictor. We consider Distributed Gradient Descent (DGD) as a representative algorithm and analytically prove CAFe's superiority to Distributed Compressed Gradient Descent (DCGD) with biased compression in the non-convex regime with bounded gradient dissimilarity. We further prove that CAFe-S converges to a stationary point, with a rate that improves as the server's data become more representative. Experimental results in FL scenarios validate the superiority of our approaches over existing compression schemes.

Figures (4)

• Figure 1: Analysis of the CAFe working principle on a synthetic logistic regression task. Left: Global training loss showing convergence. Middle: The Compression Gain Ratio ($\rho$) remains below 1.0 for the majority of training, indicating that the difference vector has a smaller norm than the raw update. The ratio approaches 1.0 as the model converges. Right: The log-density histogram shows that CAFe updates are more peaked at zero (sparser) than direct updates.
• Figure 2: Learning curves comparing direct compression (Direct) and CAFe with Low-rank compression in the CIFAR-10 task for iid and non-iid settings. CAFe results are presented with solid lines, and Direct compression results are presented with dashed lines. CAFe demonstrates faster convergence and higher accuracy.
• Figure 3: Accuracy of CAFe-S as a function of server's data representativeness ($\beta$) on CIFAR-100. While CAFe-S improves as the server's data become more representative, CAFe remains superior due to the stability of the aggregated predictor.
• Figure 4: Learning rate sensitivity analysis on CIFAR-10 (iid) with rank-3 low-rank approximation compression. Both approaches achieve optimal performance around $\gamma=10^{-1}$ and $\gamma=10^{-0.5}$ and show similar shapes, suggesting the stricter theoretical bound for CAFe may not be a practical limitation.

Theorems & Definitions (21)

• Definition 1
• Example 1: Top-k compression qsgd
• Remark 2: Communication Trade-off
• Theorem 1
• proof : Proof Sketch
• Theorem 2
• proof : Proof Sketch
• Theorem 3
• proof : Proof Sketch
• Theorem 3