FedSA: A Unified Representation Learning via Semantic Anchors for Prototype-based Federated Learning

TL;DR

FedSA addresses representation malleability in prototype-based federated learning caused by statistical and model heterogeneity by introducing semantic anchors that serve as model-agnostic prototypes. It couples anchor-based regularization (RMCL) and classifier calibration (CC) with an EMA-updated semantic anchor framework to enforce intra-class compactness, inter-class separability, and consistent decision boundaries across heterogeneous clients. The approach shifts prototype generation away from biased local representations, enabling a more unified and generalizable data representation with lightweight communication. Empirical results on CIFAR-10/100 and Tiny-Imagenet demonstrate robust gains over existing prototype-based FL methods in both cross-silo and cross-device settings, highlighting practical impact for privacy-preserving distributed learning under heterogeneity.

Abstract

Prototype-based federated learning has emerged as a promising approach that shares lightweight prototypes to transfer knowledge among clients with data heterogeneity in a model-agnostic manner. However, existing methods often collect prototypes directly from local models, which inevitably introduce inconsistencies into representation learning due to the biased data distributions and differing model architectures among clients. In this paper, we identify that both statistical and model heterogeneity create a vicious cycle of representation inconsistency, classifier divergence, and skewed prototype alignment, which negatively impacts the performance of clients. To break the vicious cycle, we propose a novel framework named Federated Learning via Semantic Anchors (FedSA) to decouple the generation of prototypes from local representation learning. We introduce a novel perspective that uses simple yet effective semantic anchors serving as prototypes to guide local models in learning consistent representations. By incorporating semantic anchors, we further propose anchor-based regularization with margin-enhanced contrastive learning and anchor-based classifier calibration to correct feature extractors and calibrate classifiers across clients, achieving intra-class compactness and inter-class separability of prototypes while ensuring consistent decision boundaries. We then update the semantic anchors with these consistent and discriminative prototypes, which iteratively encourage clients to collaboratively learn a unified data representation with robust generalization. Extensive experiments under both statistical and model heterogeneity settings show that FedSA significantly outperforms existing prototype-based FL methods on various classification tasks.

Figures (3)

• Figure 1: A toy example with two clients shows the vicious cycle. Different colors represent classes, while triangles, circles and squares represent data representations, local prototypes and global prototypes, respectively. Different colored solid lines indicate decision boundaries. Black and yellow dotted arrows show the inter-class separation among prototypes, and red arrows depict the guidance of prototypes during local training. Fig. \ref{['fig: motivation']}(a) shows that biased datasets lead feature extractors and classifiers to learn inconsistent representations and skewed decision boundaries. Fig. \ref{['fig: motivation']}(b) shows that in FedProto, the naive averaging aggregation and local guidance reduce the separability of global prototypes and inevitably create a vicious cycle.
• Figure 2: Framework of the proposed method FedSA. Pentagons represent semantic anchors. In the local training of client $m$, the semantic anchors correct the biased feature extractor and classifier via RMCL and CC, achieving intra-class compactness and inter-class separability of local prototypes, while ensuring consistent decision boundaries. On the server, we update the semantic anchors with global prototypes via an EMA update to facilitate collaboration in FL.
• Figure 3: The test accuracy (%) on Cifar100 in the cross-silo setting using the HtFE$_8$ setting under both statistical and model heterogeneity.