FedLoGe: Joint Local and Generic Federated Learning under Long-tailed Data

TL;DR

This paper introduces an approach termed Federated Local and Generic Model Training in Fed-LT (FedLoGe), which enhances both local and generic model performance through the integration of representation learning and classifier alignment within a neural collapse framework.

Abstract

Federated Long-Tailed Learning (Fed-LT), a paradigm wherein data collected from decentralized local clients manifests a globally prevalent long-tailed distribution, has garnered considerable attention in recent times. In the context of Fed-LT, existing works have predominantly centered on addressing the data imbalance issue to enhance the efficacy of the generic global model while neglecting the performance at the local level. In contrast, conventional Personalized Federated Learning (pFL) techniques are primarily devised to optimize personalized local models under the presumption of a balanced global data distribution. This paper introduces an approach termed Federated Local and Generic Model Training in Fed-LT (FedLoGe), which enhances both local and generic model performance through the integration of representation learning and classifier alignment within a neural collapse framework. Our investigation reveals the feasibility of employing a shared backbone as a foundational framework for capturing overarching global trends, while concurrently employing individualized classifiers to encapsulate distinct refinements stemming from each client's local features. Building upon this discovery, we establish the Static Sparse Equiangular Tight Frame Classifier (SSE-C), inspired by neural collapse principles that naturally prune extraneous noisy features and foster the acquisition of potent data representations. Furthermore, leveraging insights from imbalance neural collapse's classifier norm patterns, we develop Global and Local Adaptive Feature Realignment (GLA-FR) via an auxiliary global classifier and personalized Euclidean norm transfer to align global features with client preferences. Extensive experimental results on CIFAR-10/100-LT, ImageNet, and iNaturalist demonstrate the advantage of our method over state-of-the-art pFL and Fed-LT approaches.

Figures (7)

• Figure 1: (a): The mean (sorted in descending order) and variance of class means unveil feature degeneration: The feature collapse property ceases to prevail, and features with diminished means exhibit substantial variance rather than being zero; (b): After training the backbone with SSE-C, in which noisy features with bigger variance are partially pruned (gray shaded vertical lines), and enhance the quality (smaller variance) of dominant features.
• Figure 2: The framework of FedLoGe. The SSE-C enhances the capability of the backbone (feature extractor); during feature alignment, the model transfers the most crucial information from the global classifier to the personalized model, omitting information pertaining to categories with low usage.
• Figure 3: (a): Pruning features from smaller means and larger means, respectively. Negligible features exert a minor impact on model performance; (b): Pruning Experiments with SSE-C. The model optimally enhances the dominant features, rendering the impact of negligible features imperceptible.
• Figure 4: (a): The impact of varying norms $\gamma$ in SSE-C on model performance within CIFAR-100-LT; (b) The impact of sparse ratio $\beta$ on model performance in CIFAR-10-LT; (c): The impact of sparse ratio $\beta$ on model performance in CIFAR-100-LT.
• Figure 5: Accuracy of local training after GLA-FR.