Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Unified Contrastive Loss for Self-Training

Aurelien Gauffre, Julien Horvat, Massih-Reza Amini

TL;DR

This work proposes a general framework to enhance self-training methods, which replaces all instances of CE losses with a unique contrastive loss, and results in significant performance improvements across three different datasets with a limited number of labeled data.

Abstract

Self-training methods have proven to be effective in exploiting abundant unlabeled data in semi-supervised learning, particularly when labeled data is scarce. While many of these approaches rely on a cross-entropy loss function (CE), recent advances have shown that the supervised contrastive loss function (SupCon) can be more effective. Additionally, unsupervised contrastive learning approaches have also been shown to capture high quality data representations in the unsupervised setting. To benefit from these advantages in a semi-supervised setting, we propose a general framework to enhance self-training methods, which replaces all instances of CE losses with a unique contrastive loss. By using class prototypes, which are a set of class-wise trainable parameters, we recover the probability distributions of the CE setting and show a theoretical equivalence with it. Our framework, when applied to popular self-training methods, results in significant performance improvements across three different datasets with a limited number of labeled data. Additionally, we demonstrate further improvements in convergence speed, transfer ability, and hyperparameter stability. The code is available at \url{https://github.com/AurelienGauffre/semisupcon/}.

A Unified Contrastive Loss for Self-Training

TL;DR

This work proposes a general framework to enhance self-training methods, which replaces all instances of CE losses with a unique contrastive loss, and results in significant performance improvements across three different datasets with a limited number of labeled data.

Abstract

Self-training methods have proven to be effective in exploiting abundant unlabeled data in semi-supervised learning, particularly when labeled data is scarce. While many of these approaches rely on a cross-entropy loss function (CE), recent advances have shown that the supervised contrastive loss function (SupCon) can be more effective. Additionally, unsupervised contrastive learning approaches have also been shown to capture high quality data representations in the unsupervised setting. To benefit from these advantages in a semi-supervised setting, we propose a general framework to enhance self-training methods, which replaces all instances of CE losses with a unique contrastive loss. By using class prototypes, which are a set of class-wise trainable parameters, we recover the probability distributions of the CE setting and show a theoretical equivalence with it. Our framework, when applied to popular self-training methods, results in significant performance improvements across three different datasets with a limited number of labeled data. Additionally, we demonstrate further improvements in convergence speed, transfer ability, and hyperparameter stability. The code is available at \url{https://github.com/AurelienGauffre/semisupcon/}.
Paper Structure (21 sections, 11 equations, 3 figures, 3 tables)

This paper contains 21 sections, 11 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Notations and Background
  3. Notations.
  4. Supervised Contrastive Learning.
  5. Unsupervised Contrastive Learning.
  6. Self-training AminiSelf-TrainingSurvey.
  7. Method
  8. Overview
  9. Unsupervised part ${\bm{Z}}^u, {\bm{y}}^u$.
  10. Centroid part ${\bm{Z}}^c,{\bm{y}}^c$.
  11. A unified loss $\mathcal{L}_{SSC}$.
  12. Weigthed Semi-SupCon Loss
  13. Link with cross-entropy
  14. Experiments
  15. Experimental setup
  16. ...and 6 more sections

Figures (3)

  • Figure 1: SSC framework. ${\bm{Z}}^x$, ${\bm{Z}}^u$, and ${\bm{Z}}^c$ are supervised, unsupervised, and prototype embeddings and ${\bm{y}}^x$, ${\bm{y}}^u$, and ${\bm{y}}^c$ their corresponding labels, which aim to define positive pairs in the loss. Both the triplets of embeddings and their corresponding labels are concatenated and subsequently input into the loss function. The weakly augmented embeddings ${\bm{Z}}^w$ are used only during pseudo-labeling phase to compute ${\bm{y}}^u$ and does not propagate gradient back. Strongly augmented embeddings ${\bm{Z}}^u$ used two augmentations to ensure the existence of at least one positive pair for unconfident examples.
  • Figure 2: Transfer Performance with FixMatch on CIFAR-100 with 25 labels per class. The color gray (resp. green) corresponds to FixMatch without $\mathcal{L}_{SSC}$ (resp. with $\mathcal{L}_{SSC}$), while the dashed (resp. solid) line represents training from scratch (resp. using MoCo v2 weights).
  • Figure 3: Hyperparameter stability analysis with FixMatch on CIFAR-100. For each hyperparameter, 10 experiments are conducted using the same seed with different values uniformly distributed in the followings range : $\tau\in [.9,0.98]$, $\mu \in \{3,...,12\}$ and RandAugment strength parameter in {3,...,20}. All experiments are run on 256 epochs with 25 labels/class.