What We Don't C: Representations for scientific discovery beyond VAEs

TL;DR

The paper tackles the challenge of accessing meaningful features in high-dimensional scientific data that are not explicitly captured by standard variational models. It proposes latent_flow_matching with classifier-free guidance to disentangle conditioning information from residual latent structure, linking the VAE latent space to a base distribution via a velocity field and a conditioning dropout mechanism. Through experiments on 2D Gaussians, colored MNIST, and Galaxy10_DECaLS, it demonstrates that the method can selectively remove or preserve conditioning signals, enable style transfer, and isolate class-specific content in real images, thereby revealing information hidden from conventional representations. This approach offers a computationally tractable pathway for scientific discovery, enabling researchers to explore What We Don't Capture and repurpose latent representations without full retraining when conditioning information changes.

Abstract

Accessing information in learned representations is critical for scientific discovery in high-dimensional domains. We introduce a novel method based on latent flow matching with classifier-free guidance that disentangles latent subspaces by explicitly separating information included in conditioning from information that remains in the residual representation. Across three experiments -- a synthetic 2D Gaussian toy problem, colored MNIST, and the Galaxy10 astronomy dataset -- we show that our method enables access to meaningful features of high dimensional data. Our results highlight a simple yet powerful mechanism for analyzing, controlling, and repurposing latent representations, providing a pathway toward using generative models for scientific exploration of what we don't capture, consider, or catalog.

Figures (7)

• Figure 1: Using a VAE data manifold, aggregated labels can be used to remove recorded factors of variation from the latent space using a flow matching model. This enables access to features that are less apparent in the VAE manifold but are important in describing the underlying data. By disentangling learned manifolds from the information provided by labels, the resulting structure can be used to iteratively discover What We Don't C.
• Figure 2: Visualization of the 2D Gaussian experiment colored by Gaussian class index and Euclidean distance $d$ of each sample to the center of its Gaussian at $t=1$.
• Figure 3: $R^2$ scores of linear regression model trained to predict the r, g, and b values throughout the conditional and unconditional flow. Note that b is withheld and is consistently recovered throughout both flows.
• Figure 4: t-SNE projections of cMNIST embeddings from three points in the flow. The reverse conditional flow removes the most visible class structure found in the VAE latents while the unconditional flow preserves the dominant class specific structure of the VAE latents.
• Figure 5: Style transfer in colored MNIST: the conditional$t=0$ embeddings are then used with a different conditioning digit to produce stylistically similar digits in the VAE space. This demonstrates that stylistic features are captured and disentangled by the conditional distribution.