Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Self-Supervised Learning from Structural Invariance

Yipeng Zhang, Hafez Ghaemi, Jungyoon Lee, Shahab Bakhtiari, Eilif B. Muller, Laurent Charlin

TL;DR

AdaSSL introduces a latent variable to capture the structured uncertainty in one-to-many target mappings inherent to natural positive pairs in self-supervised learning. By decomposing mutual information between paired embeddings and introducing a variational (AdaSSL-V) or sparse-edit (AdaSSL-S) mechanism, it provides a tractable objective that regularizes the latent factor while leveraging standard SSL losses. Across numerical, synthetic, natural images, and video tasks, AdaSSL consistently improves the learning of content factors, disentanglement, and world-modeling capabilities, particularly under heteroscedastic or multimodal conditionals. This framework generalizes SSL to better reflect real-world data generation, enabling more robust and expressive representations with broad applicability in causal representation learning and video understanding.

Abstract

Joint-embedding self-supervised learning (SSL), the key paradigm for unsupervised representation learning from visual data, learns from invariances between semantically-related data pairs. We study the one-to-many mapping problem in SSL, where each datum may be mapped to multiple valid targets. This arises when data pairs come from naturally occurring generative processes, e.g., successive video frames. We show that existing methods struggle to flexibly capture this conditional uncertainty. As a remedy, we introduce a latent variable to account for this uncertainty and derive a variational lower bound on the mutual information between paired embeddings. Our derivation yields a simple regularization term for standard SSL objectives. The resulting method, which we call AdaSSL, applies to both contrastive and distillation-based SSL objectives, and we empirically show its versatility in causal representation learning, fine-grained image understanding, and world modeling on videos.

Self-Supervised Learning from Structural Invariance

TL;DR

AdaSSL introduces a latent variable to capture the structured uncertainty in one-to-many target mappings inherent to natural positive pairs in self-supervised learning. By decomposing mutual information between paired embeddings and introducing a variational (AdaSSL-V) or sparse-edit (AdaSSL-S) mechanism, it provides a tractable objective that regularizes the latent factor while leveraging standard SSL losses. Across numerical, synthetic, natural images, and video tasks, AdaSSL consistently improves the learning of content factors, disentanglement, and world-modeling capabilities, particularly under heteroscedastic or multimodal conditionals. This framework generalizes SSL to better reflect real-world data generation, enabling more robust and expressive representations with broad applicability in causal representation learning and video understanding.

Abstract

Joint-embedding self-supervised learning (SSL), the key paradigm for unsupervised representation learning from visual data, learns from invariances between semantically-related data pairs. We study the one-to-many mapping problem in SSL, where each datum may be mapped to multiple valid targets. This arises when data pairs come from naturally occurring generative processes, e.g., successive video frames. We show that existing methods struggle to flexibly capture this conditional uncertainty. As a remedy, we introduce a latent variable to account for this uncertainty and derive a variational lower bound on the mutual information between paired embeddings. Our derivation yields a simple regularization term for standard SSL objectives. The resulting method, which we call AdaSSL, applies to both contrastive and distillation-based SSL objectives, and we empirically show its versatility in causal representation learning, fine-grained image understanding, and world modeling on videos.
Paper Structure (76 sections, 5 theorems, 41 equations, 16 figures, 8 tables)

This paper contains 76 sections, 5 theorems, 41 equations, 16 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Problem formulation
  3. Data generating process
  4. Preliminaries: contrastive SSL
  5. Preliminaries: distillation-based SSL
  6. Unavoidable heteroscedasticity
  7. Method
  8. Modeling uncertainty with a latent variable
  9. AdaSSL-V and a lower bound on $I\mathopen{}\mathclose{\left(f({\mathbf{x}}), f({\mathbf{x}}^+)\right)$
  10. Similarity function.
  11. AdaSSL-S and sparse modular edits
  12. Experiments
  13. Overview of experimental protocol
  14. Baselines.
  15. H-InfoNCE.
  16. ...and 61 more sections

Key Result

Proposition 2.1

Let $\mathbb{S}^{d_{f}} \subset \mathbb{R}^{d_{f}+1}$ denote the $d_f$-dimensional unit sphere. Let $g:\mathbb{R}^{d_z} \to \mathbb{R}^{d_x}$ be $C^{1}$ diffeomorphic to its image, and let $f:\mathbb{R}^{d_x} \to \mathbb{S}^{d_f}$ be $C^{1}$ almost everywhere. Define $h:=f\circ g:\mathbb R^{d_z}\to\

Figures (16)

  • Figure 1: Visual comparison of models. Boxes denote vectors and arrows denote functions. We use $f$ to denote both encoders although they may use different parameters in practice. (a) Contrastive SSL typically uses a symmetric architecture. (b) Distillation-based SSL uses a predictor to predict the embeddings of one branch from the other, optionally with the help of some supervision ${\mathbf{r}}^\star$ related to the difference between the inputs. (c) Our method, AdaSSL, extends SSL by modeling the latent variable ${\mathbf{r}}$. (d) AdaSSL-V learns a variational distribution, $q_\phi({\mathbf{r}} \mid {\mathbf{x}}, {\mathbf{x}}^+)$, and uses an MLP as predictor. (e) AdaSSL-S regularizes the sparsity of ${\mathbf{r}}$ and uses a modular predictor.
  • Figure 2: Illustration of different types of noise structure in $p({\mathbf{z}}^+ \mid {\mathbf{z}})$. Here, "constant" refers to isotropic, homoscedastic noise. Conditioning on a latent variable ${\mathbf{r}}$ can transform the noise into a simpler form. For example, a car may turn left or right, producing a bimodal conditional distribution; conditioning on the driver’s intention removes the irrelevant mode.
  • Figure 3: Identifiability results on 3DIdent. AdaSSL achieves the best disentanglement and $R^2$ scores. " " denotes "same as above".
  • Figure 4: Linear $F_1$ scores on representations (encoder output) and embeddings (projector output) trained on CelebA, under weak or strong augmentations. AdaSSL+GT, a soft performance upper bound, uses the ground-truth attribute difference as ${\mathbf{r}}$.
  • Figure 5: Visualization of images paired by class label from the iNat-1M dataset.
  • ...and 11 more figures

Theorems & Definitions (9)

  • Proposition 2.1
  • Lemma B.1
  • proof
  • Proposition B.1
  • proof
  • Proposition B.2: Tangent-space variant of Proposition \ref{['thm:C1']}
  • proof
  • Proposition B.3: Extension of Proposition \ref{['thm:C1']}
  • proof