Efficient Flow Matching using Latent Variables

TL;DR

Efficient Flow Matching using Latent Variables introduces Latent-CFM, a framework that incorporates low-dimensional latent structure into conditional flow matching by conditioning the transport vector field on latent features extracted from data. It leverages pretrained lightweight latent-variable models such as VAEs or GMMs to parametrize the conditioning, enabling up to 50% fewer training steps while achieving competitive or superior sample quality on synthetic multimodal data and standard image benchmarks, as well as yielding physically consistent samples for 2D Darcy flow. A crucial advantage is conditional generation and latent-space interpretability, demonstrated via feature-conditioned sampling and compositional generation. The work highlights the potential of data-structure-aware flow learning for scalable high-dimensional generative modeling and physics-informed simulation.

Abstract

Flow matching models have shown great potential in image generation tasks among probabilistic generative models. However, most flow matching models in the literature do not explicitly utilize the underlying clustering structure in the target data when learning the flow from a simple source distribution like the standard Gaussian. This leads to inefficient learning, especially for many high-dimensional real-world datasets, which often reside in a low-dimensional manifold. To this end, we present $\texttt{Latent-CFM}$, which provides efficient training strategies by conditioning on the features extracted from data using pretrained deep latent variable models. Through experiments on synthetic data from multi-modal distributions and widely used image benchmark datasets, we show that $\texttt{Latent-CFM}$ exhibits improved generation quality with significantly less training and computation than state-of-the-art flow matching models by adopting pretrained lightweight latent variable models. Beyond natural images, we consider generative modeling of spatial fields stemming from physical processes. Using a 2d Darcy flow dataset, we demonstrate that our approach generates more physically accurate samples than competing approaches. In addition, through latent space analysis, we demonstrate that our approach can be used for conditional image generation conditioned on latent features, which adds interpretability to the generation process.

Figures (14)

• Figure 1: (a)-(d) showing generation quality of different flow-based generative models on 2d triangle dataset with 16 modes. Popular conditional flow matching approaches generated samples fail to capture the multi-modal structure of the target data distribution. (e) Latent-CFM generates samples similar to the data samples, capturing the multi-modal data structure.
• Figure 2: Schematic of Latent-CFM framework. Given a data $x_1$, Latent-CFM extracts latent features using a frozen encoder and a trainable stochastic layer. The features are embedded using a linear layer and added to the learned vector field. The framework resembles an encoder-decoder architecture like VAEs.
• Figure 3: W2 vs epochs on 2d synthetic dataset
• Figure 4: FID vs training steps show that Latent-CFM with pretrained VAE shows better generation quality compared to baseline methods on both CIFAR10 and ImageNet datasets early in the training process.
• Figure 5: Plot showing a generated sample from I-CFM (top row) and Latent-CFM (bottom row) models trained on 10K samples generated by solving Darcy Flow equations jacobsen2025cocogen. Visually generated samples from both models resemble true Pressure and Permeability fields. However, Latent-CFM exhibits a better fit to the Darcy flow equations as measured by the residuals.

Theorems & Definitions (3)

• Proposition A.1
• proof
• Lemma M.1