Quantized and Asynchronous Federated Learning

TL;DR

This paper addresses the dual challenges of communication efficiency and asynchrony in federated learning by introducing Quantized Asynchronous Federated Learning (QAFeL). QAFeL employs a hidden-state quantization scheme with buffered, buffered aggregation to prevent error propagation and enable scalable updates without requiring uniform client participation. The authors prove an optimal ergodic convergence rate of $\mathcal{O}\big(1/\sqrt{T}\big)$ for non-convex objectives, under mild assumptions, and show that the cross-term between staleness and quantization is of higher order and thus negligible. Empirically, QAFeL demonstrates strong communication savings and robust convergence on logistic regression and neural network benchmarks (CIFAR-10, CelebA, Shakespeare), outperforming or matching baselines like FedBuff while reducing communication costs.

Abstract

Recent advances in federated learning have shown that asynchronous variants can be faster and more scalable than their synchronous counterparts. However, their design does not include quantization, which is necessary in practice to deal with the communication bottleneck. To bridge this gap, we develop a novel algorithm, Quantized Asynchronous Federated Learning (QAFeL), which introduces a hidden-state quantization scheme to avoid the error propagation caused by direct quantization. QAFeL also includes a buffer to aggregate client updates, ensuring scalability and compatibility with techniques such as secure aggregation. Furthermore, we prove that QAFeL achieves an $\mathcal{O}(1/\sqrt{T})$ ergodic convergence rate for stochastic gradient descent on non-convex objectives, which is the optimal order of complexity, without requiring bounded gradients or uniform client arrivals. We also prove that the cross-term error between staleness and quantization only affects the higher-order error terms. We validate our theoretical findings on standard benchmarks.

Figures (10)

• Figure 1: Flowchart comparing synchronous and asynchronous FL.
• Figure 2: Numerical example of the effect of direct quantization (Naive) vs. the unquantized counterpart. No client quantization is performed. We show unbiased ($\mathop{\mathrm{QSGD}}\nolimits$qsgd) and biased ($\mathop{\mathrm{top-k}}\nolimits$error_feedback) server quantizer examples. We consider a logistic regression problem with $\ell_2$ regularization on the mushrooms dataset from LIBSVM LIBSVM. The simulation parameters are: 100 clients, delays following a half-normal distribution, server buffer size of 10, client learning rate of 2, server learning rate of 0.1, and $\ell_2$ regularization strength of $1/8124$, where $8124$ is the number of samples in the dataset. The $y$-axis illustrates the difference $f(x) - f^\star$, where $f(x)$ is the global model cost at a given iteration, and $f^*$ is the optimal cost.
• Figure 3: Block diagram for updating the hidden-state.
• Figure 4: QAFeL block diagram. Shaded parts occur on the client side. Dashed lines indicate communication between the clients and the server.
• Figure 5: Numerical example of the effect of naive quantization vs. our proposed algorithm (QAFeL). No client quantization is performed. The $y$-axis illustrates the difference $f(x) - f^\star$, where $f(x)$ is the global model cost at a given iteration, and $f^*$ is the optimal cost. Subfigure (a) illustrates the unbiased server quantizer case. Subfigure (b) illustrates the biased server quantizer case.

Theorems & Definitions (9)

• Definition
• Definition
• Theorem 3.7
• Corollary 3.8: QAFeL's order of complexity
• Remark A.1
• Lemma A.2
• Lemma A.3
• Corollary A.4
• Lemma A.5