CoRe-Fed: Bridging Collaborative and Representation Fairness via Federated Embedding Distillation

TL;DR

CoRe-Fed addresses fairness gaps in federated learning under data heterogeneity by targeting representation bias and collaborative bias. It combines embedding-level alignment via contrastive learning and embedding-level knowledge distillation with a dynamic, participation-aware aggregation that weights clients according to both their recent involvement and semantic alignment to the global embedding, using $w_i$ that blends $(1/f_i)^{\gamma}$ with $\sigma(k\rho_i)$. Empirical results on FMNIST and CIFAR-10 under Dirichlet non-IID partitions show that CoRe-Fed achieves higher accuracy and substantially lower representation and performance bias than strong baselines, with ablations confirming the necessity of both components. The work advances practical fair FL by linking representation coherence with equitable aggregation, and suggests future directions toward personalization-aware fairness in diverse, real-world deployments.

Abstract

With the proliferation of distributed data sources, Federated Learning (FL) has emerged as a key approach to enable collaborative intelligence through decentralized model training while preserving data privacy. However, conventional FL algorithms often suffer from performance disparities across clients caused by heterogeneous data distributions and unequal participation, which leads to unfair outcomes. Specifically, we focus on two core fairness challenges, i.e., representation bias, arising from misaligned client representations, and collaborative bias, stemming from inequitable contribution during aggregation, both of which degrade model performance and generalizability. To mitigate these disparities, we propose CoRe-Fed, a unified optimization framework that bridges collaborative and representation fairness via embedding-level regularization and fairness-aware aggregation. Initially, an alignment-driven mechanism promotes semantic consistency between local and global embeddings to reduce representational divergence. Subsequently, a dynamic reward-penalty-based aggregation strategy adjusts each client's weight based on participation history and embedding alignment to ensure contribution-aware aggregation. Extensive experiments across diverse models and datasets demonstrate that CoRe-Fed improves both fairness and model performance over the state-of-the-art baseline algorithms.

Figures (5)

• Figure 1: Illustration of representation bias in FL using CIFAR-10 automobiles images. Clients trained on biased local data, e.g., cars vs. trucks, leads to misaligned feature representations in the global embedding space. As shown in the upper section, angular divergence emerges when client representations deviate from semantically correct directions. This misalignment causes degraded performance, particularly, during inference on bias-conflicting samples.
• Figure 2: Architecture overview of CoRe-Fed framework. Clients perform local training and extract embedding vectors using DNN-based feature extractors. A server refines embedding space using contrastive learning over client and global embeddings. An alignment vector then guides knowledge distillation to adjust client representations towards a global semantic structure. Finally, contribution-aware aggregation is performed based on clients' embedding alignment and participation frequency.
• Figure 3: Mean test accuracy and fairness on CIFAR-10 with batch sizes 50 and 200; 20 out of 100 clients online per round.
• Figure 4: The average per client test accuracy on FMNIST with batch size 200; 100% of 10 clients online per round.
• Figure 5: Ablation experiments comparing three scenarios.