Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generalization Bounds for Robust Contrastive Learning: From Theory to Practice

Ngoc N. Tran, Lam Tran, Hoang Phan, Anh Bui, Tung Pham, Toan Tran, Dinh Phung, Trung Le

TL;DR

The paper addresses the theoretical gap in understanding how the unsupervised phase of contrastive learning influences robustness in the downstream supervised phase. It derives upper bounds showing that not only the adversarial InfoNCE loss but also the benign InfoNCE loss and a global divergence between benign and adversarial latent distributions contribute to robust performance, and it introduces a framework that exploits these insights. The proposed method, GSA-RSL, combines Sharpness-Aware Minimization on the benign loss with a global-divergence regularizer implemented via GANs and a discriminator, yielding improvements in both clean and robust accuracy. Empirical results on CIFAR-10/100 and STL-10 validate the theory, demonstrating enhanced robustness and transferability when all components are integrated.

Abstract

Contrastive Learning first extracts features from unlabeled data, followed by linear probing with labeled data. Adversarial Contrastive Learning (ACL) integrates Adversarial Training into the first phase to enhance feature robustness against attacks in the probing phase. While ACL has shown strong empirical results, its theoretical understanding remains limited. Furthermore, while a fair amount of theoretical works analyze how the unsupervised loss can support the supervised loss in the probing phase, none has examined its role to the robust supervised loss. To fill this gap, our work develops rigorous theories to identify which components in the unsupervised training can help improve the robust supervised loss. Specifically, besides the adversarial contrastive loss, we reveal that the benign one, along with a global divergence between benign and adversarial examples can also improve robustness. Proper experiments are conducted to justify our findings.

Generalization Bounds for Robust Contrastive Learning: From Theory to Practice

TL;DR

The paper addresses the theoretical gap in understanding how the unsupervised phase of contrastive learning influences robustness in the downstream supervised phase. It derives upper bounds showing that not only the adversarial InfoNCE loss but also the benign InfoNCE loss and a global divergence between benign and adversarial latent distributions contribute to robust performance, and it introduces a framework that exploits these insights. The proposed method, GSA-RSL, combines Sharpness-Aware Minimization on the benign loss with a global-divergence regularizer implemented via GANs and a discriminator, yielding improvements in both clean and robust accuracy. Empirical results on CIFAR-10/100 and STL-10 validate the theory, demonstrating enhanced robustness and transferability when all components are integrated.

Abstract

Contrastive Learning first extracts features from unlabeled data, followed by linear probing with labeled data. Adversarial Contrastive Learning (ACL) integrates Adversarial Training into the first phase to enhance feature robustness against attacks in the probing phase. While ACL has shown strong empirical results, its theoretical understanding remains limited. Furthermore, while a fair amount of theoretical works analyze how the unsupervised loss can support the supervised loss in the probing phase, none has examined its role to the robust supervised loss. To fill this gap, our work develops rigorous theories to identify which components in the unsupervised training can help improve the robust supervised loss. Specifically, besides the adversarial contrastive loss, we reveal that the benign one, along with a global divergence between benign and adversarial examples can also improve robustness. Proper experiments are conducted to justify our findings.
Paper Structure (14 sections, 6 theorems, 21 equations, 1 figure, 4 tables, 1 algorithm)

This paper contains 14 sections, 6 theorems, 21 equations, 1 figure, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Self-Supervised Learning
  4. Flat Minima
  5. Problem Formulation and Notions
  6. Proposed Framework
  7. Theory Development
  8. Practical Method
  9. Experiments
  10. Experimental Settings
  11. Experimental Results
  12. Main results
  13. Ablation Studies
  14. Conclusion

Key Result

Theorem 4.1

Consider the adversarial loss $\mathcal{L}^{\mathrm{adv}}_{\mathcal{D}_{\mathrm{sup}}}(\theta, \mathbf{\bar{a}})$. i) We have the first upper bound on the data space ii) We have the second upper bound on the latent space Here we note that $D_v$ represents a $f$-divergence ali1966generalnguyen2005divergences with the corresponding convex function $v(t) = (t-1)^2$.

Figures (1)

  • Figure 1: Clean & robust accuracy with/out our method's components, across different weighting for the benign loss term.

Theorems & Definitions (10)

  • Theorem 4.1
  • Theorem 4.2
  • Remark 4.3
  • Theorem 4.4
  • Theorem 4.5
  • Remark 4.6
  • Theorem 4.7
  • Remark 4.8
  • Theorem 4.9
  • Remark 4.10