Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FedUV: Uniformity and Variance for Heterogeneous Federated Learning

Ha Min Son, Moon-Hyun Kim, Tai-Myoung Chung, Chao Huang, Xin Liu

TL;DR

FedUV tackles non-IID data in federated learning by directly emulating IID conditions through two local regularizers: hyperspherical uniformity of encoder representations and classifier-variance regularization on final-layer outputs. The approach avoids reliance on global models, instead shaping internal representations and output distributions to remain IID-like across clients. Empirical results show FedUV achieving state-of-the-art performance on label-shift and feature-shift benchmarks, with faster convergence and superior efficiency compared to existing methods. These findings suggest practical, scalable improvements for robust FL under heterogeneous data distributions.

Abstract

Federated learning is a promising framework to train neural networks with widely distributed data. However, performance degrades heavily with heterogeneously distributed data. Recent work has shown this is due to the final layer of the network being most prone to local bias, some finding success freezing the final layer as an orthogonal classifier. We investigate the training dynamics of the classifier by applying SVD to the weights motivated by the observation that freezing weights results in constant singular values. We find that there are differences when training in IID and non-IID settings. Based on this finding, we introduce two regularization terms for local training to continuously emulate IID settings: (1) variance in the dimension-wise probability distribution of the classifier and (2) hyperspherical uniformity of representations of the encoder. These regularizations promote local models to act as if it were in an IID setting regardless of the local data distribution, thus offsetting proneness to bias while being flexible to the data. On extensive experiments in both label-shift and feature-shift settings, we verify that our method achieves highest performance by a large margin especially in highly non-IID cases in addition to being scalable to larger models and datasets.

FedUV: Uniformity and Variance for Heterogeneous Federated Learning

TL;DR

FedUV tackles non-IID data in federated learning by directly emulating IID conditions through two local regularizers: hyperspherical uniformity of encoder representations and classifier-variance regularization on final-layer outputs. The approach avoids reliance on global models, instead shaping internal representations and output distributions to remain IID-like across clients. Empirical results show FedUV achieving state-of-the-art performance on label-shift and feature-shift benchmarks, with faster convergence and superior efficiency compared to existing methods. These findings suggest practical, scalable improvements for robust FL under heterogeneous data distributions.

Abstract

Federated learning is a promising framework to train neural networks with widely distributed data. However, performance degrades heavily with heterogeneously distributed data. Recent work has shown this is due to the final layer of the network being most prone to local bias, some finding success freezing the final layer as an orthogonal classifier. We investigate the training dynamics of the classifier by applying SVD to the weights motivated by the observation that freezing weights results in constant singular values. We find that there are differences when training in IID and non-IID settings. Based on this finding, we introduce two regularization terms for local training to continuously emulate IID settings: (1) variance in the dimension-wise probability distribution of the classifier and (2) hyperspherical uniformity of representations of the encoder. These regularizations promote local models to act as if it were in an IID setting regardless of the local data distribution, thus offsetting proneness to bias while being flexible to the data. On extensive experiments in both label-shift and feature-shift settings, we verify that our method achieves highest performance by a large margin especially in highly non-IID cases in addition to being scalable to larger models and datasets.
Paper Structure (17 sections, 4 equations, 4 figures, 5 tables)

This paper contains 17 sections, 4 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Federated Learning (FL)
  4. Aggregation Regularization
  5. Local Training Regularization
  6. Proposed Method --- FedUV
  7. Classifier Variance
  8. Hyperspherical Uniformity
  9. Experiments and Results
  10. Experimental Setup
  11. Results
  12. Discussion
  13. Preventing Classifier Bias
  14. Hyperspherical Uniformity
  15. Convergence of FedUV
  16. ...and 2 more sections

Figures (4)

  • Figure 1: Singular values of the weights of the classifier (final layer) trained on CIFAR-100. Training setup is specified in Section \ref{['experimental_setup']}
  • Figure 2: Uniformity regularization is applied to the penultimate layer and variance regularization is applied to the output layer.
  • Figure 3: The training loss for FedAvg and FedUV across different settings
  • Figure 4: t-SNE on the features (output of the penultimate layer) of STL-10 (Class #1). Experiment setup is equivalent to main text. Clients and class were selected at random.