Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Calibre: Towards Fair and Accurate Personalized Federated Learning with Self-Supervised Learning

Sijia Chen, Ningxin Su, Baochun Li

TL;DR

Calibre tackles fair and accurate personalized federated learning in non-i.i.d. settings by calibrating self-supervised representations through client-specific prototypes and a prototype-guided aggregation mechanism. It formalizes the Generality-Personalization Tradeoff with a theorem that balances information flows via a constrained objective, implemented as a four-term loss $L^c = l_c + l_s + \alpha(l_p + l_n)$ augmented by prototype generation and client-adaptive regularizers. Empirically, Calibre achieves state-of-the-art mean accuracy and reduced fairness variance across CIFAR-10, CIFAR-100, and STL-10, and extends effectively to novel unseen clients, outperforming label-dependent and other SSL-based pFL baselines. The approach provides a practical, SSL-based framework for reliable personalization in FL by preserving fairness while leveraging unlabeled data and lightweight per-client heads, with potential impact on privacy-preserving collaborative learning in heterogeneous environments.

Abstract

In the context of personalized federated learning, existing approaches train a global model to extract transferable representations, based on which any client could train personalized models with a limited number of data samples. Self-supervised learning is considered a promising direction as the global model it produces is generic and facilitates personalization for all clients fairly. However, when data is heterogeneous across clients, the global model trained using SSL is unable to learn high-quality personalized models. In this paper, we show that when the global model is trained with SSL without modifications, its produced representations have fuzzy class boundaries. As a result, personalized learning within each client produces models with low accuracy. In order to improve SSL towards better accuracy without sacrificing its advantage in fairness, we propose Calibre, a new personalized federated learning framework designed to calibrate SSL representations by maintaining a suitable balance between more generic and more client-specific representations. Calibre is designed based on theoretically-sound properties, and introduces (1) a client-specific prototype loss as an auxiliary training objective; and (2) an aggregation algorithm guided by such prototypes across clients. Our experimental results in an extensive array of non-i.i.d.~settings show that Calibre achieves state-of-the-art performance in terms of both mean accuracy and fairness across clients. Code repo: https://github.com/TL-System/plato/tree/main/examples/ssl/calibre.

Calibre: Towards Fair and Accurate Personalized Federated Learning with Self-Supervised Learning

TL;DR

Calibre tackles fair and accurate personalized federated learning in non-i.i.d. settings by calibrating self-supervised representations through client-specific prototypes and a prototype-guided aggregation mechanism. It formalizes the Generality-Personalization Tradeoff with a theorem that balances information flows via a constrained objective, implemented as a four-term loss augmented by prototype generation and client-adaptive regularizers. Empirically, Calibre achieves state-of-the-art mean accuracy and reduced fairness variance across CIFAR-10, CIFAR-100, and STL-10, and extends effectively to novel unseen clients, outperforming label-dependent and other SSL-based pFL baselines. The approach provides a practical, SSL-based framework for reliable personalization in FL by preserving fairness while leveraging unlabeled data and lightweight per-client heads, with potential impact on privacy-preserving collaborative learning in heterogeneous environments.

Abstract

In the context of personalized federated learning, existing approaches train a global model to extract transferable representations, based on which any client could train personalized models with a limited number of data samples. Self-supervised learning is considered a promising direction as the global model it produces is generic and facilitates personalization for all clients fairly. However, when data is heterogeneous across clients, the global model trained using SSL is unable to learn high-quality personalized models. In this paper, we show that when the global model is trained with SSL without modifications, its produced representations have fuzzy class boundaries. As a result, personalized learning within each client produces models with low accuracy. In order to improve SSL towards better accuracy without sacrificing its advantage in fairness, we propose Calibre, a new personalized federated learning framework designed to calibrate SSL representations by maintaining a suitable balance between more generic and more client-specific representations. Calibre is designed based on theoretically-sound properties, and introduces (1) a client-specific prototype loss as an auxiliary training objective; and (2) an aggregation algorithm guided by such prototypes across clients. Our experimental results in an extensive array of non-i.i.d.~settings show that Calibre achieves state-of-the-art performance in terms of both mean accuracy and fairness across clients. Code repo: https://github.com/TL-System/plato/tree/main/examples/ssl/calibre.
Paper Structure (18 sections, 3 theorems, 6 equations, 8 figures, 1 table, 1 algorithm)

This paper contains 18 sections, 3 theorems, 6 equations, 8 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Formulation
  4. Model Fairness and Overall Performance
  5. Personalized FL with Self-Supervised Learning
  6. Representations with Fuzzy Class Boundaries
  7. Calibre: a New pFL Framework
  8. Generality-Personalization Tradeoff Theorem
  9. Design
  10. Experimental Results
  11. Experimental Setup
  12. Accuracy
  13. Fairness
  14. Performance on Novel Clients
  15. Rationale Behind the Superior Performance of Calibre (SimCLR)
  16. ...and 3 more sections

Key Result

Lemma 1

Given $\bm{\theta}_b$ and the learned representation $\overline{\bm{z}}$, we can implicitly encourage the model to use the client's local training data $D$ by maximizing the lower bound of $I(\overline{y}^{\prime};D|\bm{\theta}_b,\overline{\bm{z}})$ shown as $I(\bm{x}^{\prime};\overline{y}^{\prime}|

Figures (8)

  • Figure 1: Illustrations of 2D t-SNE embeddings of SSL representations learned by four methods. Moving from left to right, the representations are derived from local samples of 10 out of 100 clients, using encoders trained by pFL-SimCLR and pFL-BYOL respectively.
  • Figure 2: Visualization of 2D t-SNE embeddings of client representations and test accuracy derived from pFL-SimCLR and pFL-BYOL methods. The first three representations are from pFL-SimCLR, while the last three are from pFL-BYOL. These examples are randomly selected from a pool of 100 clients.
  • Figure 3: Comparison of Mean and Variance of Test Accuracy among $100$ clients across different non-i.i.d. settings in the CIFAR-10, CIFAR-100, and STL-10 datasets.
  • Figure 4: Comparison of Mean and Variance of test accuracy of 150 clients in CIFAR-10 and CIFAR-100 datasets under the distribution-based label non-i.i.d..
  • Figure 5: Illustrations of 2D t-SNE embeddings for representations obtained from encoders trained with pFL-SimSiam, pFL-MoCoV2, Calibre (SimSiam), and Calibre (MoCoV2), respectively. This experiment is conducted on the CIFAR-10 dataset under D-non-i.i.d. with the concentration $0.3$. We collect representations from $6$ out of $100$ clients for visualization.
  • ...and 3 more figures

Theorems & Definitions (6)

  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Theorem 1: Generality-Personalization Tradeoff
  • proof