An Inclusive Theoretical Framework of Robust Supervised Contrastive Loss against Label Noise

TL;DR

This work introduces a unified theoretical framework for robust supervised contrastive losses under label noise, deriving a general condition that links the noisy and clean risks for arbitrary contrastive losses. It demonstrates that the standard InfoNCE loss is non-robust and proposes SymNCE, a robust counterpart formed by adding a Reverse InfoNCE term, supported by theoretical risk analysis. The framework is shown to encompass existing robust techniques such as nearest-neighbor sample selection and the RINCE loss, and is validated by extensive experiments on CIFAR-10/100, Tiny Imagenet, and Clothing1M, where SymNCE consistently improves robustness and accuracy under various noise regimes. Overall, the paper provides both rigorous guarantees and a practical robust loss design to improve learning from noisy labels in supervised contrastive settings.

Abstract

Learning from noisy labels is a critical challenge in machine learning, with vast implications for numerous real-world scenarios. While supervised contrastive learning has recently emerged as a powerful tool for navigating label noise, many existing solutions remain heuristic, often devoid of a systematic theoretical foundation for crafting robust supervised contrastive losses. To address the gap, in this paper, we propose a unified theoretical framework for robust losses under the pairwise contrastive paradigm. In particular, we for the first time derive a general robust condition for arbitrary contrastive losses, which serves as a criterion to verify the theoretical robustness of a supervised contrastive loss against label noise. The theory indicates that the popular InfoNCE loss is in fact non-robust, and accordingly inspires us to develop a robust version of InfoNCE, termed Symmetric InfoNCE (SymNCE). Moreover, we highlight that our theory is an inclusive framework that provides explanations to prior robust techniques such as nearest-neighbor (NN) sample selection and robust contrastive loss. Validation experiments on benchmark datasets demonstrate the superiority of SymNCE against label noise.

Figures (2)

• Figure 1: Comparisons in linear probing accuracy between supervised contrastive learning (SupCon), unsupervised contrastive learning (SimCLR), and robust supervised contrastive learning (SymNCE).
• Figure 2: (a)(b) Parameter analysis of weight parameter $\beta$ in SymNCE under symmetric and asymmetric label noise. (c) Verification of theoretical choice of $\lambda$ in RINCE.

Theorems & Definitions (18)

• Lemma 3.2
• proof : Proof of Lemma \ref{['lem::pi']}
• Lemma 3.3
• proof : Proof Sketch of Lemma \ref{['lem::decom']}
• Theorem 3.4
• proof : Proof Sketch of Theorem \ref{['thm::riskconsistency']}
• Theorem 3.5
• proof : Proof of Theorem \ref{['thm::noisetolerant']}
• Theorem 4.1
• proof : Proof of Theorem \ref{['prop::RevNCE']}