Generative Modeling of Discrete Data Using Geometric Latent Subspaces

TL;DR

The paper addresses generative modeling of high-dimensional discrete data by introducing Geometric PCA (GPCA) latent subspaces embedded in the exponential parameter space and endowed with an \(e\)-metric. By constructing an isometric link between the latent subspace and the data manifold \(\mathcal{M}=\partial\psi(\mathcal{U})\), it enables straight-line \(e\)-geodesics and efficient flow matching for learning generative models that capture statistical dependencies. Empirical results on MNIST, Cityscapes, and DNA datasets demonstrate that small latent dimensions suffice for accurate reconstruction and competitive generation, highlighting substantial compression without severe performance loss. The approach offers a principled, geometry-aware framework for discrete data generation with scalable training, while outlining avenues for improvement via OT-based enhancements and broader isometry analyses.

Abstract

We introduce the use of latent subspaces in the exponential parameter space of product manifolds of categorial distributions, as a tool for learning generative models of discrete data. The low-dimensional latent space encodes statistical dependencies and removes redundant degrees of freedom among the categorial variables. We equip the parameter domain with a Riemannian geometry such that the spaces and distances are related by isometries which enables consistent flow matching. In particular, geodesics become straight lines which makes model training by flow matching effective. Empirical results demonstrate that reduced latent dimensions suffice to represent data for generative modeling.

Figures (13)

• Figure 1: GPCA- vs. PCA-subspaces.(a) The hypercube $\mathcal{X}=\mathbb{H}^{3}=\{0,1\}^{3}$ as discrete data space. Any$d=2$-dimensional affine subspace, when used with linear projection and rounding, can represent at most $2^{d}=4$ points exactly. (b) A $2$-dimensional geometric latent subspace $\mathcal{U}$ represents $6>4$ data points exactly using real latent coordinates. (c) The nonlinear manifold $\mathcal{M}$ induced by the linear GPCA-subspace $\mathcal{U}$ in the data space. (d) Since (a) is isotropic, its covariance is proportional to the identity, so PCA does not reveal a dominant direction. Dropping one coordinate and projection yields the 4 points $\{(1/2,x_{2},x_{3}) : x_{2},x_{3}\in\{0,1\}\}$, which are non-integral and do not belong to $\mathcal{X}$.
• Figure 2: Approach (sketch): Transport of a reference measure on a low-dimensional geometric subspace $\mathcal{U}$ implicitly learns statistical dependencies and a generative model of discrete data, 'spanned' after decoding as extreme points of (the closure of) a nonlinear data manifold $\mathcal{M}=\partial\psi(\mathcal{U})$, as illustrated by Figure \ref{['fig:H3-GPCA']}.
• Figure 3: A 2D embedding of the $d=30$ dimensional latent points $\theta_{i} = V z_{i}\in\mathcal{U},\, i\in[N]$ corresponding to $10.000$ samples $x_{i}\approx \partial\psi(\theta_{i})$ of the MNIST data base (see Figure \ref{['fig:GPCA-embedding-3D']} in Appendix \ref{['sec:app-additional-graphics']} for a more detailed 3D embedding). This semantically meaningful and fully unsupervised embedding of the $4$-nearest neighbor graph with nodes $\theta_{i}$ is purely geometric: no learning is involved besides determining the GPCA subspace $\mathcal{U}$. This result underlines the suitability of latent GPCA representations of discrete data, as illustrated by Figure \ref{['fig:H3-GPCA']}. See Figure \ref{['fig:MNIST-GPCA-samples']} in Appendix \ref{['sec:app-additional-graphics']} for a sample of GPCA-based data point approximations.
• Figure 4: (a) The submanifold of all factorizing discrete distributions of two binary variables inside the set of all joint distributions represented as tetrahedron in local coordinates. Examples of foliations of all joint distributions by 2D GPCA subspaces (b)-(d) and by 1D GPCA subspaces (e)-(f). Comparison to (a) shows that learning GPCA subspaces entails learning statistical dependencies of discrete random variables.
• Figure 5: Reconstruction error measured by the Hamming distance (Eq. \ref{['eq:hamming-recon-error']}). The $y$-axis uses an inverse hyperbolic sine scaling $y'=\mathrm{sinh^{-1}}(y/0.05)$. Shaded regions indicate the minimum and maximum reconstruction errors over the training set.

Theorems & Definitions (14)

• Definition 3.1: $e$-metric
• Proposition 3.2: $\nabla^e = \nabla^1$
• Remark 3.3: geodesics
• Proposition 3.4: isometry relations
• Proposition 3.5: geodesic interpolants on $\mathcal{M}$ versus $\mathcal{S}_{c}^{n}$
• Proposition 4.1: CFM in natural parameters
• Remark 4.2: $e$-metric versus Fisher-Rao metric
• Remark 4.3: pullback metric on $\mathcal{Z}$
• Corollary 4.4: approximate flow matching
• proof : Proof of Proposition \ref{['prop:e-connection']}