Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dynamically Scaled Temperature in Self-Supervised Contrastive Learning

Siladittya Manna, Soumitri Chattopadhyay, Rakesh Dey, Saumik Bhattacharya, Umapada Pal

TL;DR

This work tackles the distribution-shaping role of the temperature parameter in InfoNCE-based self-supervised contrastive learning by introducing DySTreSS, a dynamic temperature scaling framework that maps cosine similarity $s_{ij}$ to a per-pair temperature $\tau_{ij}$ via $\tau_{ij} = \tau_{min} + \tfrac{1}{2}(\tau_{max}-\tau_{min})\bigl(1 + \cos(\pi(1 + s_{ij}))\bigr)$. The authors provide theoretical insights into how temperature modulates penalties for false negatives and hard negatives and show that a cosine-based scaling satisfies criteria designed to preserve local structure, avoid excessive FN penalties, and maintain global convergence. Empirically, DySTreSS yields consistent improvements over strong SSL baselines (e.g., SimCLR, MACL) on ImageNet1K/100, CIFAR10/100, and long-tailed variants, and demonstrates strong transfer performance in vision and sentence-embedding tasks, with favorable Uniformity and Inter-Class Uniformity characteristics. Overall, the work establishes a principled, similarity-aware mechanism to adaptively regulate the InfoNCE loss, enhancing representation quality and transferability in diverse settings.

Abstract

In contemporary self-supervised contrastive algorithms like SimCLR, MoCo, etc., the task of balancing attraction between two semantically similar samples and repulsion between two samples of different classes is primarily affected by the presence of hard negative samples. While the InfoNCE loss has been shown to impose penalties based on hardness, the temperature hyper-parameter is the key to regulating the penalties and the trade-off between uniformity and tolerance. In this work, we focus our attention on improving the performance of InfoNCE loss in self-supervised learning by proposing a novel cosine similarity dependent temperature scaling function to effectively optimize the distribution of the samples in the feature space. We also provide mathematical analyses to support the construction of such a dynamically scaled temperature function. Experimental evidence shows that the proposed framework outperforms the contrastive loss-based SSL algorithms.

Dynamically Scaled Temperature in Self-Supervised Contrastive Learning

TL;DR

This work tackles the distribution-shaping role of the temperature parameter in InfoNCE-based self-supervised contrastive learning by introducing DySTreSS, a dynamic temperature scaling framework that maps cosine similarity to a per-pair temperature via . The authors provide theoretical insights into how temperature modulates penalties for false negatives and hard negatives and show that a cosine-based scaling satisfies criteria designed to preserve local structure, avoid excessive FN penalties, and maintain global convergence. Empirically, DySTreSS yields consistent improvements over strong SSL baselines (e.g., SimCLR, MACL) on ImageNet1K/100, CIFAR10/100, and long-tailed variants, and demonstrates strong transfer performance in vision and sentence-embedding tasks, with favorable Uniformity and Inter-Class Uniformity characteristics. Overall, the work establishes a principled, similarity-aware mechanism to adaptively regulate the InfoNCE loss, enhancing representation quality and transferability in diverse settings.

Abstract

In contemporary self-supervised contrastive algorithms like SimCLR, MoCo, etc., the task of balancing attraction between two semantically similar samples and repulsion between two samples of different classes is primarily affected by the presence of hard negative samples. While the InfoNCE loss has been shown to impose penalties based on hardness, the temperature hyper-parameter is the key to regulating the penalties and the trade-off between uniformity and tolerance. In this work, we focus our attention on improving the performance of InfoNCE loss in self-supervised learning by proposing a novel cosine similarity dependent temperature scaling function to effectively optimize the distribution of the samples in the feature space. We also provide mathematical analyses to support the construction of such a dynamically scaled temperature function. Experimental evidence shows that the proposed framework outperforms the contrastive loss-based SSL algorithms.
Paper Structure (33 sections, 18 equations, 8 figures, 14 tables, 3 algorithms)

This paper contains 33 sections, 18 equations, 8 figures, 14 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Theoretical Background
  5. Role of Temperature in Contrastive Learning
  6. Effect of Temperature on Local and Global Structures
  7. Motivation of Proposed Temperature Scaling Function
  8. Proposed Framework
  9. Implementation Details
  10. Experimental Results and Ablations
  11. Comparison with SSL Frameworks on ImageNet100
  12. Comparison with SSL Frameworks on ImageNet1K
  13. Comparison on Small Scale Benchmarks
  14. Experimental Results on Long-Tailed Datasets
  15. Comparison with Negative-free Contrastive and Non-Contrastive Frameworks
  16. ...and 18 more sections

Figures (8)

  • Figure 1: (a) Histogram of cosine similarities of true positive (TP), false negative (FN), and true negative (TN) pairs at random initialization, (b) Histogram of cosine similarities of TP, FN, and TN pairs after pre-training. (Best viewed at 300%)
  • Figure 2: Plots of the solution of ODE in Eqn. \ref{['eqn:tauij']} for different values of the integral constant, over different values of $\delta$ and $K$.
  • Figure 3: Temperature functions for different $\tau_{max}$ and $\tau_{min}$. Best visible at 200%.
  • Figure 4: Plot of shifted versions of Temperature functions. Best visible at 200%.
  • Figure 5: Plot of Uniformity and Tolerance vs. 20NN Top-1 acc. shown in Table \ref{['tab:in100_bal_bs256_lars_200eps_temprange_abl']}. Best visible at 300%.
  • ...and 3 more figures