A Theoretical Analysis of Contrastive Unsupervised Representation Learning

TL;DR

This work provides a theoretical framework for contrastive unsupervised representation learning by positing latent data classes that generate similar pairs and negatives. It derives generalization bounds connecting unsupervised loss to downstream supervised performance, highlights the role of intraclass concentration and mean classifiers, and analyzes the effects of the number of negatives and block-based similarities (CURL). The authors prove guarantees for one and multiple negative samples, discuss the limitations of negative sampling, and validate key ideas with controlled text and image experiments. The results offer guidance on when contrastive methods yield transferable representations and motivate block-based extensions for stronger guarantees and practical gains.

Abstract

Recent empirical works have successfully used unlabeled data to learn feature representations that are broadly useful in downstream classification tasks. Several of these methods are reminiscent of the well-known word2vec embedding algorithm: leveraging availability of pairs of semantically "similar" data points and "negative samples," the learner forces the inner product of representations of similar pairs with each other to be higher on average than with negative samples. The current paper uses the term contrastive learning for such algorithms and presents a theoretical framework for analyzing them by introducing latent classes and hypothesizing that semantically similar points are sampled from the same latent class. This framework allows us to show provable guarantees on the performance of the learned representations on the average classification task that is comprised of a subset of the same set of latent classes. Our generalization bound also shows that learned representations can reduce (labeled) sample complexity on downstream tasks. We conduct controlled experiments in both the text and image domains to support the theory.

Figures (2)

• Figure 1: In both examples we have uniform distribution over classes ${\mathcal{C}}=\{c_1,c_2\}$, blue and red points are in $c_1$ and $c_2$ respectively and ${\mathcal{D}}_{c_i}$ is uniform over the points of $c_i$. In the first figure we have one point per class, while in the second we have two points per class. Let ${\mathcal{F}}=\{f_0,f_1\}$ where $f_0$ maps all points to $(0,0)$ and $f_1$ is defined in the figure. In both cases, using the hinge loss, $L_{sup}(f_1)=0$, $L_{sup}(f_0) = 1$ and in the second case $L_{sup}^\mu(f_1)=0$. However, in both examples the algorithm will pick $f_0$ since $L_{un}(f_0) = 1$ but $L_{un}(f_1) = \Omega(r^2)$.
• Figure 2: Effect of amount of unlabeled data and # of negative samples on unsupervised representations, measured on binary classification for CIFAR100 in (a) and on top-1 performance on Wiki-3029 in Fig (b) (top-1 performance is used because avg binary was same for all $k$). Fig. (c) shows the dynamics of train/test loss; supervised loss roughly tracks unsupervised test loss, as suggested by Theorem \ref{['thm:unsup_upper_bound']}

Theorems & Definitions (25)

• Definition 2.1: Mean Classifier
• Definition 2.2: Average Supervised Loss
• Definition 2.3: Unsupervised Loss
• Theorem 4.1
• Remark
• Lemma 4.2
• Lemma 4.3
• proof
• proof : Proof of Theorem \ref{['thm:unsup_upper_bound']}
• Lemma 4.4