FedAdaVR: Adaptive Variance Reduction for Robust Federated Learning under Limited Client Participation

TL;DR

This work tackles partial client participation in federated learning by introducing FedAdaVR, a server-side adaptive optimiser combined with a SAGA-like variance reduction that uses stored updates from inactive clients to emulate full participation, thereby reducing variance due to client unavailability. It further proposes FedAdaVR-Quant, a memory-efficient variant that stores updates in quantised form (FP16/Int8/Int4) with substantial memory reductions up to 87.5% while preserving performance. The authors provide a nonconvex convergence guarantee showing elimination of partial participation error and demonstrate through extensive experiments on IID and non-IID partitions across vision and NLP tasks that FedAdaVR and FedAdaVR-Quant consistently outperform state-of-the-art baselines. The methods achieve faster convergence and robust performance with minimal extra burden on clients, offering practical improvements for robust, scalable federated learning under limited participation.

Abstract

Federated learning (FL) encounters substantial challenges due to heterogeneity, leading to gradient noise, client drift, and partial client participation errors, the last of which is the most pervasive but remains insufficiently addressed in current literature. In this paper, we propose FedAdaVR, a novel FL algorithm aimed at solving heterogeneity issues caused by sporadic client participation by incorporating an adaptive optimiser with a variance reduction technique. This method takes advantage of the most recent stored updates from clients, even when they are absent from the current training round, thereby emulating their presence. Furthermore, we propose FedAdaVR-Quant, which stores client updates in quantised form, significantly reducing the memory requirements (by 50%, 75%, and 87.5%) of FedAdaVR while maintaining equivalent model performance. We analyse the convergence behaviour of FedAdaVR under general nonconvex conditions and prove that our proposed algorithm can eliminate partial client participation error. Extensive experiments conducted on multiple datasets, under both independent and identically distributed (IID) and non-IID settings, demonstrate that FedAdaVR consistently outperforms state-of-the-art baseline methods.

Figures (5)

• Figure 1: Accuracy Comparison Across Various Data Partitioning Methods—IID, IID-NonIID, Dirichlet, LQ-1, LQ-2, and LQ-3 (Left to Right)—on MNIST (Top), FMNIST (Middle), and CIFAR-10 (Bottom) Datasets.
• Figure 2: Ablation study analysing the impact of variance reduction and the adaptive optimiser. The plots compare accuracy (left) and loss (right) across FedAdaVR, FedAdaVR-NoOPT, and FedAdaVR-NoVR on the CIFAR-10 dataset (LQ-1 partitioning).
• Figure 3: Loss Comparison Across Different Data Partitioning Methods—IID, IID-NonIID, Dirichlet, LQ-1, LQ-2, and LQ-3 (From Left to Right)—on MNIST (Top), FMNIST (Middle), and CIFAR-10 (Bottom) Datasets.
• Figure 4: Accuracy and loss comparison on Shakespeare dataset (Natural ID Partitioner). Note that Scaffold and MIFA are omitted from the loss plots due to excessive out-of-range fluctuations.
• Figure 5: Ablation study on the impact of variance reduction and optimiser. The plots compare accuracy and loss across FedAdaVR, FedAdaVR-NoOPT, and FedAdaVR-NoVR on CIFAR-10 dataset (LQ-1 partitioning).

Theorems & Definitions (10)

• Theorem 5.4: Convergence of FedAdaVR
• Corollary 5.5
• Lemma 1.1: Young’s inequality
• Lemma 1.2: Jensen’s inequality
• Lemma 1.3: Sum of squares
• Lemma 1.4: Variance under uniform, without‐replacement sampling
• Lemma 1.5
• Lemma 1.6
• Lemma 1.7
• proof