Contrastive Diffusion Alignment: Learning Structured Latents for Controllable Generation

TL;DR

Diffusion models offer high-fidelity generation but lack explicit dynamics-aware latent structure. ConDA introduces a dual-space framework that learns a compact contrastive embedding $\mathcal{C}$ of diffusion latents $\mathcal{Z}$ using auxiliary variables (e.g., time, stimulation), enabling nonlinear trajectory editing in $\mathcal{C}$ while rendering in $\mathcal{Z}$ for fidelity. The method combines a conditional latent diffusion model in $\mathcal{Z}$ with a contrastively learned, dynamics-aligned $\mathcal{C}$; editors such as spline and Taylor extrapolation enable interpolation, extrapolation, and class transfers with faithful reconstructions. Across five spatiotemporal domains (fluid dynamics, neural calcium imaging, therapeutic neurostimulation, facial expressions, and motor control), ConDA yields superior controllability and temporal coherence compared to linear traversals and conditioning baselines, demonstrating that diffusion latents encode dynamics-relevant structure that can be exploited via latent organization and manifold traversal for practical controllable generation.

Abstract

Diffusion models excel at generation, but their latent spaces are not explicitly organized for interpretable control. We introduce ConDA (Contrastive Diffusion Alignment), a framework that applies contrastive learning within diffusion embeddings to align latent geometry with system dynamics. Motivated by recent advances showing that contrastive objectives can recover more disentangled and structured representations, ConDA organizes diffusion latents such that traversal directions reflect underlying dynamical factors. Within this contrastively structured space, ConDA enables nonlinear trajectory traversal that supports faithful interpolation, extrapolation, and controllable generation. Across benchmarks in fluid dynamics, neural calcium imaging, therapeutic neurostimulation, and facial expression, ConDA produces interpretable latent representations with improved controllability compared to linear traversals and conditioning-based baselines. These results suggest that diffusion latents encode dynamics-relevant structure, but exploiting this structure requires latent organization and traversal along the latent manifold.

Figures (15)

• Figure 1: Method overview of ConDA framework. We encode a high-dimensional sequence into a cLDM feature latent via DDIM inversion, then map these latents into a compact, contrastively learned space that organizes local geometry to reflect intrinsic dynamics. An editing operator modifies the trajectory under new conditions (local traversal or class transfer). The edited embedding is lifted back to the feature space (via a learned decoder or neighborhood interpolation) and decoded by the cLDM’s conditional generator to produce the new output frames.
• Figure 2: Visual comparison of generated images from ConDA-Spline/TEX-2 with baselines. Spline and TEX-2 vs. linear (Lerp) and nonlinear (LSTM) baselines, showing superior perceptual quality and dynamics-consistent reconstructions for Spline and TEX-2.
• Figure 2: Baseline comparison: ConDA vs linear and variational baselines. Performance metrics for test sequence prediction comparing ConDA to PCA and $\beta$-VAE. ConDA achieves improved performance indicating closer alignment with the underlying dynamical manifold.
• Figure 3: SVM classification performance and KDE based class interpolation. (Left) Metrics for classifying steady vs. unsteady laminar flow and E-field responder vs. non-responder using $\mathcal{Z}$ and $\mathcal{C}$-space. (Right) Trajectory morphing a non-responder E-Field ($\bullet$) towards a responder ($\times$) by interpolating between class-conditional KDE peaks; the color bar shows per-pixel E-field change from the source, highlighting cortical shifts needed for responder-like patterns and suggesting a path to personalized targeting.
• Figure 4: Comparison of facial expressions at test frames. ConDA captures facial expressions (e.g., lip-corner pull/stretch) at test frames with higher fidelity than PCA and $\beta$-VAE.