FedORGP: Guiding Heterogeneous Federated Learning with Orthogonality Regularization on Global Prototypes

TL;DR

FedORGP tackles the dual challenges of data and model heterogeneity in federated learning by introducing orthogonality regularization on global class prototypes to create an angular margin that complements cross-entropy loss. The method defines intra-class prototype similarity and inter-class separation via a learned prototype system and optimizes a regularized loss L_OR to maximize intra-class compactness while expanding inter-class margins, then aligns local embeddings with the global prototypes through L_k. The authors provide convergence guarantees under non-convex conditions and demonstrate through extensive experiments on CIFAR-10/100, Flowers102, and TinyImagenet that FedORGP achieves up to 10.12 percentage points higher accuracy than strong baselines, with robustness to both statistical and model heterogeneity and varying participation rates. The results indicate that angular-prototype separation integrated with CE loss offers a practical, scalable improvement for heterogeneous FL systems.

Abstract

Federated Learning (FL) has emerged as an essential framework for distributed machine learning, especially with its potential for privacy-preserving data processing. However, existing FL frameworks struggle to address statistical and model heterogeneity, which severely impacts model performance. While Heterogeneous Federated Learning (HtFL) introduces prototype-based strategies to address the challenges, current approaches face limitations in achieving optimal separation of prototypes. This paper presents FedORGP, a novel HtFL algorithm designed to improve global prototype separation through orthogonality regularization, which not only encourages intra-class prototype similarity but also significantly expands the inter-class angular separation. With the guidance of the global prototype, each client keeps its embeddings aligned with the corresponding prototype in the feature space, promoting directional independence that integrates seamlessly with the cross-entropy (CE) loss. We provide theoretical proof of FedORGP's convergence under non-convex conditions. Extensive experiments demonstrate that FedORGP outperforms seven state-of-the-art baselines, achieving up to 10.12\% accuracy improvement in scenarios where statistical and model heterogeneity coexist.

Figures (4)

• Figure 1: We train models on the CIFAR-10 dataset and use t-SNE t-sne to visualize their performance on previously unseen test samples (16 per class) within the feature space, with triangles indicating prototypes and circles denoting samples. The results indicate that FedProto tan2022fedproto exhibits weak feature separation. FedTGP fedtgp increases prototype margin but still lacks sufficient distinction in the feature space. In contrast, our FedORGP can reduce inter-sample similarity and increase intra-class compactness, thereby effectively classifying the majority of samples. This result suggests that FedORGP exhibits robust generalization performance under conditions of both statistical and model heterogeneity.
• Figure 2: Orthogonality-constrained prototypes can expand inter-class margins while enhancing intra-class clustering.
• Figure 3: This image illustrates the FedORGP framework involving two clients. From ① to ④: We input the samples, get the representations, then input them into the classifier for prediction, and finally calculate the CE loss. ⑤: The dissimilarity between the feature representation and its corresponding prototype is calculated as a regularization term. ⑥: After the local model is updated, we collect the client prototypes. ⑦: Orthogonality regularization are applied to the global prototypes on the server to enhance angular separation.
• Figure 4: Test on Cifar-100 (10 classes per client) under different feature dimensions with HMG$_8$.