The Geometric Mechanics of Contrastive Representation Learning: Alignment Potentials, Entropic Dispersion, and Cross-Modal Divergence

TL;DR

The paper develops a measure-theoretic framework for contrastive representation learning on a fixed embedding manifold, revealing a geometric bifurcation between unimodal and multimodal regimes. In the large-batch, low-temperature limit, stochastic InfoNCE optimization converges to deterministic energy landscapes, with unimodal learning yielding a strictly convex Gibbs energy and entropy acting as a tie-breaker, while multimodal learning exhibits a persistent negative symmetric divergence that erects barriers and enforces a population-level modality gap. It further shows that under sufficient encoder expressiveness, parametric learning inherits these intrinsic geometries, providing a mechanistic explanation for phenomena like Mind the Gap and shifting emphasis from pointwise discrimination to population geometry. The results offer principled diagnostics and objective-design guidance to diagnose and mitigate distributional misalignment, enabling more robust cross-modal representations and transferability.

Abstract

While InfoNCE powers modern contrastive learning, its geometric mechanisms remain under-characterized beyond the canonical alignment--uniformity decomposition. We present a measure-theoretic framework that models learning as the evolution of representation measures on a fixed embedding manifold. By establishing value and gradient consistency in the large-batch limit, we bridge the stochastic objective to explicit deterministic energy landscapes, uncovering a fundamental geometric bifurcation between the unimodal and multimodal regimes. In the unimodal setting, the intrinsic landscape is strictly convex with a unique Gibbs equilibrium; here, entropy acts merely as a tie-breaker, clarifying "uniformity" as a constrained expansion within the alignment basin. In contrast, the symmetric multimodal objective contains a persistent negative symmetric divergence term that remains even after kernel sharpening. We show that this term induces barrier-driven co-adaptation, enforcing a population-level modality gap as a structural geometric necessity rather than an initialization artifact. Our results shift the analytical lens from pointwise discrimination to population geometry, offering a principled basis for diagnosing and controlling distributional misalignment.

Figures (8)

• Figure 1: The geometric bifurcation of contrastive representation learning. We unify unimodal and multimodal analysis under a measure-theoretic framework. Via a shared analytical path, we bridge the stochastic objective to a deterministic energy landscape in the large-batch limit ($N\to\infty$). However, at the intrinsic limit ($\tau \downarrow 0^+$), the geometries bifurcate. The unimodal landscape is governed by a strictly convex functional where entropy acts as a unique tie-breaker. Conversely, the multimodal landscape is structurally coupled by a negative symmetric divergence; this erects repulsive barriers that enforce the "modality gap" as a necessary geometric state.
• Figure 2: Large-batch gradient consistency. Comparison of batch gradients against the large-batch reference across various $N$. Left: Cosine alignment (higher is better). Right: Relative gradient error (lower is better). Results are mean $\pm$ standard error over 20 seeds.
• Figure 3: Unimodal potential landscape on $\mathbb{S}^2$ and equilibria across temperature $\tau$. Left: the two-well potential $U$ in \ref{['eq:uni_s2_potential']} (colored by value), with minima centers $\mathbf{m}_1,\mathbf{m}_2$ marked. Right: Gibbs samples (blue; importance-resampled) and trained particles (orange; minimizing \ref{['eq:uni_s2_particle_obj']}) across various temperatures. As $\tau$ decreases, both distributions concentrate around the low-energy wells.
• Figure 4: Low-$\tau$ concentration on $\mathbb{S}^2$ (mean $\pm$ std over 20 seeds). Cap mass $\mathrm{CapMass}_\varepsilon$ with $\varepsilon\!=\!0.50$ around potential wells $\{\mathbf{m}_1,\mathbf{m}_2\}$ increases as $\tau$ decreases, quantifying the ground-state trend.
• Figure 5: Marginal gap vs. misalignment (mean $\pm$ standard error). Symmetric KL divergence $D_{\mathrm{KL}}^{\mathrm{sym}}$ increases as latent misalignment $\sigma_{\mathrm{mis}}$ grows.

Theorems & Definitions (56)

• Remark 2.1: Generality of \ref{['ass:kernel_volume']}
• Definition 2.1: Population partition function
• Definition 2.2: Smoothed representation density
• Proposition 3.1: Stable unimodal optimization
• proof
• Definition 3.1: Unimodal alignment potential field
• Definition 3.2: Unimodal parametric energy
• Theorem 3.1: Large-batch unimodal dynamics
• proof
• Remark 3.1: From alignment--uniformity to representation geometry