Multi-modal contrastive learning adapts to intrinsic dimensions of shared latent variables

TL;DR

Analysis of theoretical properties of the learned representations from multi-modal contrastive learning reveals that, enabled by temperature optimization, multi-modal contrastive learning not only maximizes mutual information between modalities but also adapts to intrinsic dimensions of data, which can be much lower than user-specified dimensions for representation vectors.

Abstract

Multi-modal contrastive learning as a self-supervised representation learning technique has achieved great success in foundation model training, such as CLIP~\citep{radford2021learning}. In this paper, we study the theoretical properties of the learned representations from multi-modal contrastive learning beyond linear representations and specific data distributions. Our analysis reveals that, enabled by temperature optimization, multi-modal contrastive learning not only maximizes mutual information between modalities but also adapts to intrinsic dimensions of data, which can be much lower than user-specified dimensions for representation vectors. Experiments on both synthetic and real-world datasets demonstrate the ability of contrastive learning to learn low-dimensional and informative representations, bridging theoretical insights and practical performance.

Figures (14)

• Figure 1: Multi-modal contrastive learning applied to the bone marrow single-cell CITE-seq data.
• Figure 2: Histograms of out-of-sample similarities, change of intrinsic dimension, and convergence of temperature (linear setting: $k^* = 2$, $d = 3$).
• Figure 3: Results with synthetic data: linear setting.
• Figure 4: Results with synthetic data: nonlinear setting.
• Figure 5: Results with CITE-seq dataset.

Theorems & Definitions (22)

• theorem 1: Informal
• definition 1
• definition 2: Maximal mutual information
• definition 3: Intrinsic dimension.
• proposition 1
• theorem 2
• lemma 1
• lemma 2
• lemma 3
• lemma 4