Incremental Generation is Necessary and Sufficient for Universality in Flow-Based Modelling

TL;DR

The paper addresses why incremental generation is essential for universal flow-based modeling within the class of orientation-preserving homeomorphisms on [0,1]^d. It proves a negative result: autonomous, single-step flows cannot achieve universality, while incremental flows built from ReLU MLP vector fields are universal, with explicit rates and a finite bound on the number of flows. By lifting the domain to a higher dimension, the authors also obtain structured universal approximations for arbitrary Lipschitz functions and for probability measures under 1-Wasserstein error, linking to optimal transport via Benamou-Brenier and Caffarelli regularity. Collectively, these results provide a rigorous approximation-theoretic foundation for incremental, flow-based generation and its lifted extensions, with implications for stable, geometry-aware generative modeling and measure transport.

Abstract

Incremental flow-based denoising models have reshaped generative modelling, but their empirical advantage still lacks a rigorous approximation-theoretic foundation. We show that incremental generation is necessary and sufficient for universal flow-based generation on the largest natural class of self-maps of $[0,1]^d$ compatible with denoising pipelines, namely the orientation-preserving homeomorphisms of $[0,1]^d$. All our guarantees are uniform on the underlying maps and hence imply approximation both samplewise and in distribution. Using a new topological-dynamical argument, we first prove an impossibility theorem: the class of all single-step autonomous flows, independently of the architecture, width, depth, or Lipschitz activation of the underlying neural network, is meagre and therefore not universal in the space of orientation-preserving homeomorphisms of $[0,1]^d$. By exploiting algebraic properties of autonomous flows, we conversely show that every orientation-preserving Lipschitz homeomorphism on $[0,1]^d$ can be approximated at rate $O(n^{-1/d})$ by a composition of at most $K_d$ such flows, where $K_d$ depends only on the dimension. Under additional smoothness assumptions, the approximation rate can be made dimension-free, and $K_d$ can be chosen uniformly over the class being approximated. Finally, by linearly lifting the domain into one higher dimension, we obtain structured universal approximation results for continuous functions and for probability measures on $[0,1]^d$, the latter realized as pushforwards of empirical measures with vanishing $1$-Wasserstein error.

Figures (4)

• Figure 1: Visualizing Incremental Flow-Based Generation \ref{['eq:incrimental']}: In a denoising-type, flow-based incremental generator, an image $x$ (left) is mapped to noise by the invertible sequence $\varphi_T^{-1}\circ\cdots\circ\varphi_1^{-1}$. The model is trained to undo this via $\varphi_1\circ\cdots\circ\varphi_T$. At inference, a noise sample $Z$ is injected and propagated through $\varphi_1,\dots,\varphi_T$ to synthesize an image . Here, $T=2$. Each arrow depicts the vector field—visualized in the panel directly below—that induces one step of the incremental flow. By contrast, a non-incremental generator attempts a single-shot mapping from left to right, while non-denoising pipelines (e.g., GANs goodfellow2014generative) do not enforce invertibility.
• Figure 2: Why Non-Incremental Generation is Not Universal: The reason why non-incremental generators (\ref{['thrm:NoFlow']}) fail to be universal is that most homeomorphisms cannot be represented as flows (\ref{['thrm:NonEmbedability']}). The idea is that there is a dense open set of orientation-preserving homeomorphisms supported on the hypercube $[0,1]^d$, which can be approximated/perturbed so that any given orbit becomes periodic (SubFigure \ref{['fig:negative_result_proof_sketch__C0closing']}). Then, these perturbations can be further perturbed so that the a small neighbourhood around the given orbit becomes a basin of attraction (SubFigure \ref{['fig:negative_result_proof_sketch__Shrinking']}), which cannot happen for any flow. Consequently, the complement of any such map, which contains the set $\mathrm{NODE}_{\sigma}((0,1)^d)$ cannot be dense, implying that all non-incremental generators/autonomous Neural ODEs fail to be universal approximators of orientation-preserving homeomorphisms supported on the hypercube $[0,1]^d$.
• Figure 3: By (a uniform version) of Thursten's Theorem thurston1974foliations we $\operatorname{\mathcal{H}}_d([0,1]^d)$ is a simple group and since the group generated by flows is normal, then every diffeomorphism (green curve) must be the composition of finitely many flows of vector fields $V^{(1)}$, $\dots$, $V^{(T)})$ (here $T=4$) -- Sub-Figure \ref{['fig:positive_result__ss:decomposition']}. Each vector field is then approximated by a ReLU MLP at an optimal rate (Sub-Figure \ref{['fig:positive_result__ss:approximation']}) with maximal Lipschitz regularity using Hong_2024__ReLUMLPs; the approximation of the original homeomorphism (\ref{['thrm:Universality_of_Neural_ODEs_RH']}) is concluded using Grönwall's inequality. Furthermore, in dimension $d\ge 5$, every orientation preserving homeomorphism is isotopic to the identity, then muller2014uniform implies that it can be uniformly approximated by diffeomorphisms; reducing (\ref{['thrm:General_Universality']}) to the smooth case.
• Figure 4: Any continuous function $f:\mathbb{R}^d\to \mathbb{R}$ can be realized as a time-$1$ flow $\Phi(1,(x,y))$ for the $(d+1)$-dimensional vector field $V(x,y)\stackrel{\hbox{\upshape\tiny def.}}{=} (0,f(x))$ (illustrated by the pink vector fields), which acts trivially in its first “dummy’’ coordinates and acts as the target function $f(x)$ in the $(d+1)^{\mathrm{st}}$ coordinate. By mapping any given input $x\in \mathbb{R}^d$ to the initial condition $(x,0)\in \mathbb{R}^{d+1}$ of the flow $\Phi$ “lifting’’ $f$, we then simply flow linearly relative to the $(d+1)^{\mathrm{st}}$ (here $y$) axis until it arrives at $(x,f(x))$ at time $1$, at which point the final value can be linearly projected-off and the value $f(x)$ is recovered. In this way, every real-valued continuous (resp. Lipschitz, resp. smooth) function can be realized as a time-$1$ flow of the same regularity in a space of only one more dimension.

Theorems & Definitions (29)

• Example 1
• Example 2: A $1d$ Example of (Non-)Orientation Preserving Homeomorphism
• Example 3: Orientation-Preserving Homeomorphisms From Computer Vision (Rotations)
• Theorem 1: Incremental Generation is Necessary for Universality
• Theorem 2: Non-Incremental Generation on $[\delta,1-\delta]^d$ is meagre
• Theorem 3: Few $C^0$ Homeomorphisms are Flowable on $[0,1]^d$ for $d>1$
• Remark 4
• Theorem 5: Universal Approximation of Orientation-Preserving Homeomorphisms
• Remark 6
• Proposition 7: Non-triviality