FedDPC : Handling Data Heterogeneity and Partial Client Participation in Federated Learning

TL;DR

FedDPC tackles data heterogeneity and partial client participation in federated learning by using orthogonal projection of each local update onto the previous global update, followed by adaptive scaling of the resulting residual. This variance-reducing approach keeps local updates aligned with the global trajectory and stabilizes aggregation, enabling faster convergence and better generalization. Empirical results on heterogeneously partitioned image datasets show FedDPC outperforming state-of-the-art methods in training efficiency and test accuracy, with modest server-side overhead. The method offers a practical, scalable enhancement for real-world FL deployments and opens avenues for extension to vertical federated learning.

Abstract

Data heterogeneity is a significant challenge in modern federated learning (FL) as it creates variance in local model updates, causing the aggregated global model to shift away from the true global optimum. Partial client participation in FL further exacerbates this issue by skewing the aggregation of local models towards the data distribution of participating clients. This creates additional variance in the global model updates, causing the global model to converge away from the optima of the global objective. These variances lead to instability in FL training, which degrades global model performance and slows down FL training. While existing literature primarily focuses on addressing data heterogeneity, the impact of partial client participation has received less attention. In this paper, we propose FedDPC, a novel FL method, designed to improve FL training and global model performance by mitigating both data heterogeneity and partial client participation. FedDPC addresses these issues by projecting each local update onto the previous global update, thereby controlling variance in both local and global updates. To further accelerate FL training, FedDPC employs adaptive scaling for each local update before aggregation. Extensive experiments on image classification tasks with multiple heterogeneously partitioned datasets validate the effectiveness of FedDPC. The results demonstrate that FedDPC outperforms state-of-the-art FL algorithms by achieving faster reduction in training loss and improved test accuracy across communication rounds.

Figures (7)

• Figure 1: This figure shows the effect of client-drift in FL. Due to client-drift, the global model is optimized at the optima ($\textbf{w}$) of a different objective $\Bar{F}$ instead of the optima ($\textbf{w}^*$) of the global objective $F$ and each local model $\textbf{w}_i$ is optimized away from the global optima as well as optima of others client's objective $F_j$, where $i \neq j$.
• Figure 2: This figure describes the orthogonal projection of a local update onto the previous FL iteration's global update and use of this projection to correct this local update.
• Figure 3: Comparison of communication rounds among various methods: Training Loss and Test Accuracy for CIFAR10 image classification using LeNet5 in both the heterogeneous FL settings.
• Figure 4: Comparison of communication rounds among various methods: Training Loss and Test Accuracy for CIFAR100 image classification using Resnet18 in both the heterogeneous FL settings.
• Figure 5: Comparison of communication rounds among various methods: Training Loss and Test Accuracy for Tiny ImageNet image classification using Resnet18 in both the heterogeneous FL settings.