Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Probabilistic Model for Non-Contrastive Learning

Maximilian Fleissner, Pascal Esser, Debarghya Ghoshdastidar

TL;DR

The paper introduces a probabilistic latent-variable model for self-supervised learning and shows that the MLE for the embedding operator $W$ reduces to either PCA or a simple non-contrastive loss depending on augmentation structure. In isotropic-noise regimes, the MLE recovers the top-$k$ eigenvectors of $S_x$, while in orthogonal-noise regimes, the objective aligns with minimizing or maximizing traces that yield the top eigenvectors of $-S_\Delta$, linking SSL directly to a classical statistical framework. The authors provide theoretical derivations and numerical simulations, including a Gaussian mixture extension, demonstrating when SSL can surpass PCA and how augmentations influence the recovered latent directions. This work offers a principled lens to understand when SSL provides additional structure beyond traditional unsupervised methods and suggests directions for incorporating uncertainty and priors into SSL representations.

Abstract

Self-supervised learning (SSL) aims to find meaningful representations from unlabeled data by encoding semantic similarities through data augmentations. Despite its current popularity, theoretical insights about SSL are still scarce. For example, it is not yet known whether commonly used SSL loss functions can be related to a statistical model, much in the same as OLS, generalized linear models or PCA naturally emerge as maximum likelihood estimates of an underlying generative process. In this short paper, we consider a latent variable statistical model for SSL that exhibits an interesting property: Depending on the informativeness of the data augmentations, the MLE of the model either reduces to PCA, or approaches a simple non-contrastive loss. We analyze the model and also empirically illustrate our findings.

A Probabilistic Model for Non-Contrastive Learning

TL;DR

The paper introduces a probabilistic latent-variable model for self-supervised learning and shows that the MLE for the embedding operator reduces to either PCA or a simple non-contrastive loss depending on augmentation structure. In isotropic-noise regimes, the MLE recovers the top- eigenvectors of , while in orthogonal-noise regimes, the objective aligns with minimizing or maximizing traces that yield the top eigenvectors of , linking SSL directly to a classical statistical framework. The authors provide theoretical derivations and numerical simulations, including a Gaussian mixture extension, demonstrating when SSL can surpass PCA and how augmentations influence the recovered latent directions. This work offers a principled lens to understand when SSL provides additional structure beyond traditional unsupervised methods and suggests directions for incorporating uncertainty and priors into SSL representations.

Abstract

Self-supervised learning (SSL) aims to find meaningful representations from unlabeled data by encoding semantic similarities through data augmentations. Despite its current popularity, theoretical insights about SSL are still scarce. For example, it is not yet known whether commonly used SSL loss functions can be related to a statistical model, much in the same as OLS, generalized linear models or PCA naturally emerge as maximum likelihood estimates of an underlying generative process. In this short paper, we consider a latent variable statistical model for SSL that exhibits an interesting property: Depending on the informativeness of the data augmentations, the MLE of the model either reduces to PCA, or approaches a simple non-contrastive loss. We analyze the model and also empirically illustrate our findings.
Paper Structure (8 sections, 3 theorems, 21 equations, 3 figures)

This paper contains 8 sections, 3 theorems, 21 equations, 3 figures.

Table of Contents

  1. Introduction
  2. A probabilistic model for self-supervision
  3. Maximum Likelihood Estimates
  4. Numerical Simulation
  5. Theoretical Model
  6. Gaussian Mixture Model
  7. Related Work
  8. Discussion and Open Questions

Key Result

Lemma 1

Suppose we draw $n$ positive pairs i.i.d. from the generative process eq:model_def, in the sense that $n$ independent ground truth samples $z_i$ are generated. Define the matrix Then, the log-likelihood $L(W)$ of observing $\{(x_i,x_i^+)\}_{i=1}^n$ is maximized by minimizing where we let

Figures (3)

  • Figure 1: Model Illustration. Data $x$ is generated from a latent Gaussian, which is mapped linearly to a higher-dimensional space. Then, positive samples $x^+$ are generated as Gaussians conditioned on $x$. Depending on the covariance structure, we either recover an isotropic noise model (above) or the orthogonal noise model (below). In the latter, the positive pairs lie in a subspace orthogonal to the underlying signal direction.
  • Figure 2: Numerical analysis of the theoretical setting. Plotted is $\mathcal{L}_{PCA}-\mathcal{L}_{SSL}$ (therefore positive values imply the SSL model recovers the true embedding better then PCA), the averaged over $1000$ initializations. (left) orthogonal noise model for varying $\rho,\gamma\in(1.01,2)$. (right) isotropic nose model for varying $\sigma^2,\epsilon^2\in(0.01,1)$.
  • Figure 3: Orthogonal noise model with GMM latent space. (left) plotted is the density (under a Gaussian kernel estimate) of the true latent space in green and the representation by PCA in blue and SSL (according to Proposition \ref{['theo:ortho']}) in red. (right) Plotted are the data (in purple) and positive samples (in orange) in $\mathbb{R}^2$.

Theorems & Definitions (7)

  • Lemma 1
  • proof
  • Proposition 2
  • proof
  • Proposition 3
  • proof
  • Remark 4