Local Gradient Regulation Stabilizes Federated Learning under Client Heterogeneity

TL;DR

This paper addresses the instability of federated learning under non-IID data by identifying local gradient distortions as the main drift source. It introduces ECGR, a swarm-inspired, client-side gradient re-aggregation that separates local gradients into convergent and exploratory components, preserves their magnitude, and re-weights their contributions before standard FedAvg aggregation. The authors provide theoretical justification that ECGR reduces deviation from the true global gradient and demonstrate consistent stabilization and modest accuracy gains across image classification and medical imaging tasks, including the LC25000 dataset with ResNet-18. Importantly, ECGR requires no extra communication and remains compatible with existing FL algorithms, offering a practical, broadly applicable tool for heterogeneous deployments.

Abstract

Federated learning (FL) enables collaborative model training across distributed clients without sharing raw data, yet its stability is fundamentally challenged by statistical heterogeneity in realistic deployments. Here, we show that client heterogeneity destabilizes FL primarily by distorting local gradient dynamics during client-side optimization, causing systematic drift that accumulates across communication rounds and impedes global convergence. This observation highlights local gradients as a key regulatory lever for stabilizing heterogeneous FL systems. Building on this insight, we develop a general client-side perspective that regulates local gradient contributions without incurring additional communication overhead. Inspired by swarm intelligence, we instantiate this perspective through Exploratory--Convergent Gradient Re-aggregation (ECGR), which balances well-aligned and misaligned gradient components to preserve informative updates while suppressing destabilizing effects. Theoretical analysis and extensive experiments, including evaluations on the LC25000 medical imaging dataset, demonstrate that regulating local gradient dynamics consistently stabilizes federated learning across state-of-the-art methods under heterogeneous data distributions.

Figures (12)

• Figure 1: Framework of the ECGR strategy. (a) Illustration of swarm intelligence in honeybees: foraging paths typically consist of both chaotic and stable directions, with the stable direction dominating collective behavior. (b) Inspired by swarm intelligence, local gradients in FL are categorized into exploratory gradients and convergent gradients, which are re-aggregated such that convergent gradients dominate the resulting update. (c) A two-dimensional visualization of aggregated gradients, illustrating how ECGR reduces gradient deviation induced by data heterogeneity.
• Figure 2: Global model testing accuracy curves for different FL algorithms across multiple datasets. Each row corresponds to one algorithm, and each column presents the results on a particular dataset. The green solid line indicates the accuracy trajectory of the ECGR-extended variant, whereas the remaining curves represent the corresponding standard baselines. Each plot shows the mean testing accuracy along with the upper and lower bounds, computed from runs using 5 different random seeds.
• Figure 3: Ablation studies on CIFAR-10 with respect to learning rate $\eta$, data heterogeneity level $\alpha$, and the ECGR damping coefficient $\beta$. All curves report the mean test accuracy over five independent runs with random seeds 0, 1, 42, 999, and 2025.
• Figure 4: Visualization of per-round gradient selection under ECGR on the CIFAR-10 dataset. (a) A global 3D overview illustrating gradient selection patterns across all clients. Each cube represents a selected gradient at a specific index (x-axis) and training round (y-axis), with the client dimension encoded along the z-axis. (b) A detailed view of the selection behavior for a representative node, Client 1.
• Figure 5: Representative patches from the LC25000 dataset. From left to right: Colon Adenocarcinoma, Colon Benign Tissue, Lung Adenocarcinoma, Lung Benign Tissue, and Lung Squamous Cell Carcinoma.

Theorems & Definitions (6)

• Theorem 1
• Theorem 2: Gradient Magnitude Preservation
• Definition 1: Gradient Notation
• Definition 2: Directional Consistency
• Lemma 1: Directional Consistency Monotonicity Lemma
• Theorem 3: ECGR Error Reduction Theorem