PaddingFlow: Improving Normalizing Flows with Padding-Dimensional Noise

TL;DR

PaddingFlow tackles two core NF limitations—manifold-valued targets and discrete data—by adding padding-dimensional noise, forming x'=(x+εd, εp) that augments dimension without altering the data distribution. The method preserves tractability by using z'~N(0,I_{d+p}) and projecting to the original $d$ dimensions for generation, while modifying the ELBO to account for padding. Empirically, PaddingFlow improves unconditional density estimation on tabular and VAE-processed image datasets, and enhances conditional density estimation in IK tasks, outperforming FFJORD and SoftFlow in most benchmarks. The approach is simple to implement, computationally cheap, and broadly applicable to both discrete and continuous NF models, with code available for replication.

Abstract

Normalizing flow is a generative modeling approach with efficient sampling. However, Flow-based models suffer two issues: 1) If the target distribution is manifold, due to the unmatch between the dimensions of the latent target distribution and the data distribution, flow-based models might perform badly. 2) Discrete data might make flow-based models collapse into a degenerate mixture of point masses. To sidestep such two issues, we propose PaddingFlow, a novel dequantization method, which improves normalizing flows with padding-dimensional noise. To implement PaddingFlow, only the dimension of normalizing flows needs to be modified. Thus, our method is easy to implement and computationally cheap. Moreover, the padding-dimensional noise is only added to the padding dimension, which means PaddingFlow can dequantize without changing data distributions. Implementing existing dequantization methods needs to change data distributions, which might degrade performance. We validate our method on the main benchmarks of unconditional density estimation, including five tabular datasets and four image datasets for Variational Autoencoder (VAE) models, and the Inverse Kinematics (IK) experiments which are conditional density estimation. The results show that PaddingFlow can perform better in all experiments in this paper, which means PaddingFlow is widely suitable for various tasks. The code is available at: https://github.com/AdamQLMeng/PaddingFlow.

Figures (7)

• Figure 1: Implementation of PaddingFlow for training flow-based models. $d$ denotes the dimension of the data distribution, and $p$ denotes the dimension of padding-dimensional noise.
• Figure 2: Two ways of implementing PaddingFlow on a VAE model: 1) adding PaddingFlow noise (green lines), and 2) PaddingFlow reparameterization (blue lines). Images ($\mathcal{I}^{'}$) shown in the figure are reconstructed by the PaddingFlow-based VAE model trained on Caltech 101 Silhouettes.
• Figure 3: Comparison of FFJORD, SoftFlow, and PaddingFlow on 4 2-D distributions including 2 unconditional distributions (circles, and sines) and 2 conditional distributions (conditional circles, and conditional sines).
• Figure 4: Negative log-likelihood continuously decreases while all four metrics for evaluating IK solutions have been well-converged.
• Figure 5: Comparison of VAE models based on FFJORD, and PaddingFlow on MNIST and Frey Faces.