Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Relaxed Contrastive Learning for Federated Learning

Seonguk Seo, Jinkyu Kim, Geeho Kim, Bohyung Han

TL;DR

This work tackles gradient inconsistency in federated learning caused by client data heterogeneity by linking local gradient deviations to the distribution of feature representations. It shows that supervised contrastive learning (SCL) can mitigate deviations but induces representation collapse, hindering transferability. To address this, the authors introduce FedRCL, a relaxed supervised contrastive loss with a divergence penalty and a multi-level extension that promotes diversity across intermediate representations, thereby improving transferability and convergence. Empirical results across CIFAR-10/100 and Tiny-ImageNet under diverse non-iid settings demonstrate that FedRCL outperforms strong baselines and remains robust under low participation and varying backbones, with seamless compatibility with server-side optimization techniques. The approach offers a practical, privacy-preserving way to enhance collaborative learning in heterogeneous FL environments.

Abstract

We propose a novel contrastive learning framework to effectively address the challenges of data heterogeneity in federated learning. We first analyze the inconsistency of gradient updates across clients during local training and establish its dependence on the distribution of feature representations, leading to the derivation of the supervised contrastive learning (SCL) objective to mitigate local deviations. In addition, we show that a naïve adoption of SCL in federated learning leads to representation collapse, resulting in slow convergence and limited performance gains. To address this issue, we introduce a relaxed contrastive learning loss that imposes a divergence penalty on excessively similar sample pairs within each class. This strategy prevents collapsed representations and enhances feature transferability, facilitating collaborative training and leading to significant performance improvements. Our framework outperforms all existing federated learning approaches by huge margins on the standard benchmarks through extensive experimental results.

Relaxed Contrastive Learning for Federated Learning

TL;DR

This work tackles gradient inconsistency in federated learning caused by client data heterogeneity by linking local gradient deviations to the distribution of feature representations. It shows that supervised contrastive learning (SCL) can mitigate deviations but induces representation collapse, hindering transferability. To address this, the authors introduce FedRCL, a relaxed supervised contrastive loss with a divergence penalty and a multi-level extension that promotes diversity across intermediate representations, thereby improving transferability and convergence. Empirical results across CIFAR-10/100 and Tiny-ImageNet under diverse non-iid settings demonstrate that FedRCL outperforms strong baselines and remains robust under low participation and varying backbones, with seamless compatibility with server-side optimization techniques. The approach offers a practical, privacy-preserving way to enhance collaborative learning in heterogeneous FL environments.

Abstract

We propose a novel contrastive learning framework to effectively address the challenges of data heterogeneity in federated learning. We first analyze the inconsistency of gradient updates across clients during local training and establish its dependence on the distribution of feature representations, leading to the derivation of the supervised contrastive learning (SCL) objective to mitigate local deviations. In addition, we show that a naïve adoption of SCL in federated learning leads to representation collapse, resulting in slow convergence and limited performance gains. To address this issue, we introduce a relaxed contrastive learning loss that imposes a divergence penalty on excessively similar sample pairs within each class. This strategy prevents collapsed representations and enhances feature transferability, facilitating collaborative training and leading to significant performance improvements. Our framework outperforms all existing federated learning approaches by huge margins on the standard benchmarks through extensive experimental results.
Paper Structure (39 sections, 2 theorems, 14 equations, 18 figures, 9 tables, 1 algorithm)

This paper contains 39 sections, 2 theorems, 14 equations, 18 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Federated learning
  4. Contrastive learning in FL
  5. Preliminaries
  6. Problem setup
  7. Supervised contrastive learning
  8. Relaxed Supervised Contrastive Learning
  9. Benefit of SCL for local training
  10. Representation collapse in FL with SCL
  11. Relaxed contrastive loss for FL
  12. Multi-level contrastive training
  13. Discussion
  14. Experiment
  15. Experimental setup
  16. ...and 24 more sections

Key Result

Proposition 1

If $D(\mathbf{x}) \ll 1$, the local updates of the parameters in classification layer, $\{\Delta \psi_{y}\}_{y \in \mathcal{Y}}$, are prone to deviate from the desirable direction, i.e., $\Delta \psi_{r}\phi(\mathbf{x}) < 0$ and $\exists j \neq r$ such that $\Delta \psi_{j}\phi(\mathbf{x}) > 0$ for

Figures (18)

  • Figure 1: Tiny-ImageNet
  • Figure 2: CIFAR-100
  • Figure 4: Accuracy
  • Figure 5: Within-class variance
  • Figure 6: Between-class variance
  • ...and 13 more figures

Theorems & Definitions (5)

  • Definition 1: Sample-wise deviation bound
  • Proposition 1
  • Definition 2: Effective rank
  • Definition 1: Sample-wise deviation bound
  • Proposition 1