Kuramoto-FedAvg: Using Synchronization Dynamics to Improve Federated Learning Optimization under Statistical Heterogeneity

TL;DR

Kuramoto-FedAvg tackles client drift in non-IID federated learning by treating each client update as an oscillator phase and weighting updates according to phase alignment with the global direction. The method derives a synchronization-based aggregation rule that yields a tighter convergence bound than standard FedAvg and demonstrates faster convergence and higher accuracy on benchmarks like MNIST, FMNIST, and CIFAR-10. Theoretical results show a reduced drift term $\Gamma_{\mathrm{Kuramoto}}(t)$ leading to accelerated progress, while empirical results confirm improved stability and performance without increasing local computation or communication. This work highlights synchronization-based coordination as a practical, lightweight approach to managing gradient diversity in realistic federated settings.

Abstract

Federated learning on heterogeneous (non-IID) client data experiences slow convergence due to client drift. To address this challenge, we propose Kuramoto-FedAvg, a federated optimization algorithm that reframes the weight aggregation step as a synchronization problem inspired by the Kuramoto model of coupled oscillators. The server dynamically weighs each client's update based on its phase alignment with the global update, amplifying contributions that align with the global gradient direction while minimizing the impact of updates that are out of phase. We theoretically prove that this synchronization mechanism reduces client drift, providing a tighter convergence bound compared to the standard FedAvg under heterogeneous data distributions. Empirical validation supports our theoretical findings, showing that Kuramoto-FedAvg significantly accelerates convergence and improves accuracy across multiple benchmark datasets. Our work highlights the potential of coordination and synchronization-based strategies for managing gradient diversity and accelerating federated optimization in realistic non-IID settings.

Figures (3)

• Figure 1: Comparison of Kuramoto-FedAvg with standard FedAvg and SCAFFOLD under varying levels of statistical heterogeneity (controlled by the number of data shards per client, $s \in \{3, 5, 10\}$. A lower variance indicates better synchronization among local client models. Across all benchmark datasets, Kuramoto-FedAvg converges faster and consistently maintains lower variance throughout training.
• Figure 2: Mean test accuracy across varying degrees of statistical heterogeneity. While all methods achieve higher accuracy with reduced heterogeneity, Kuramoto-FedAvg consistently outperforms the baselines across all heterogeneity settings and datasets.
• Figure 3: Comparison of test accuracy for Kuramoto-FedAvg, FedAvg, and SCAFFOLD over several training iterations on three benchmark datasets, evaluated at different heterogeneity levels. Kuramoto-FedAvg consistently achieves higher accuracy faster than FedAvg and SCAFFOLD. The rapid accuracy improvement highlights the practical benefits of reducing gradient misalignment through synchronization, demonstrating how Kuramoto-FedAvg effectively addresses model complexity and dataset-specific optimization challenges across varying degrees of client data skewness.

Theorems & Definitions (3)

• Theorem 1: FedAvg Convergence Bound
• proof
• proof