Towards Performance-Enhanced Model-Contrastive Federated Learning using Historical Information in Heterogeneous Scenarios

TL;DR

This work tackles federated learning under coexisting data and participation heterogeneity by proposing PMFL, which fuses a history-informed model-contrastive loss at the node level with an adaptive, participation-aware aggregation strategy that leverages historical global models. The key contributions are (i) a model-contrastive term that uses multiple historical local representations to stabilize contrastive points, (ii) adaptive aggregation weights based on cumulative participation to reduce bias, and (iii) a global-model smoothing mechanism employing a sliding buffer of past global models. Empirical results on SVHN, CIFAR10, CINIC, and CIFAR100 show PMFL outperforms state-of-the-art baselines across diverse heterogeneity patterns, with improved convergence stability and robustness. The approach offers a practical pathway to reliable FL deployments in real-world, non-IID, and intermittently participating environments, and suggests further exploration of model heterogeneity and broader data distributions.

Abstract

Federated Learning (FL) enables multiple nodes to collaboratively train a model without sharing raw data. However, FL systems are usually deployed in heterogeneous scenarios, where nodes differ in both data distributions and participation frequencies, which undermines the FL performance. To tackle the above issue, this paper proposes PMFL, a performance-enhanced model-contrastive federated learning framework using historical training information. Specifically, on the node side, we design a novel model-contrastive term into the node optimization objective by incorporating historical local models to capture stable contrastive points, thereby improving the consistency of model updates in heterogeneous data distributions. On the server side, we utilize the cumulative participation count of each node to adaptively adjust its aggregation weight, thereby correcting the bias in the global objective caused by different node participation frequencies. Furthermore, the updated global model incorporates historical global models to reduce its fluctuations in performance between adjacent rounds. Extensive experiments demonstrate that PMFL achieves superior performance compared with existing FL methods in heterogeneous scenarios.

Figures (14)

• Figure 1: Test accuracy of global model in FL under varying participation rates on CIFAR10 krizhevsky2009learning using average aggregation in the Non-IID setting. Results are shown for two model architectures: (a) CNN and (b) ResNet18.
• Figure 2: Visualization of class distributions and participation frequencies across 20 nodes.
• Figure 3: Visualization of various participation patterns $\mathbb{E}[p_k]= 0.1$ over the first 400 Rounds for a single node.
• Figure 4: Training accuracy of PMFL and the compared methods across different training rounds.
• Figure 5: Node-wise distributions of training accuracy and loss value on SVHN with Bernoulli participation.

Theorems & Definitions (2)

• Definition 1
• Remark 1