Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning normalizing flows from Entropy-Kantorovich potentials

Chris Finlay, Augusto Gerolin, Adam M Oberman, Aram-Alexandre Pooladian

TL;DR

This formulation allows us to train a dual objective comprised only of the scalar potential functions, and removes the burden of explicitly computing normalizing flows during training.

Abstract

We approach the problem of learning continuous normalizing flows from a dual perspective motivated by entropy-regularized optimal transport, in which continuous normalizing flows are cast as gradients of scalar potential functions. This formulation allows us to train a dual objective comprised only of the scalar potential functions, and removes the burden of explicitly computing normalizing flows during training. After training, the normalizing flow is easily recovered from the potential functions.

Learning normalizing flows from Entropy-Kantorovich potentials

TL;DR

This formulation allows us to train a dual objective comprised only of the scalar potential functions, and removes the burden of explicitly computing normalizing flows during training.

Abstract

We approach the problem of learning continuous normalizing flows from a dual perspective motivated by entropy-regularized optimal transport, in which continuous normalizing flows are cast as gradients of scalar potential functions. This formulation allows us to train a dual objective comprised only of the scalar potential functions, and removes the burden of explicitly computing normalizing flows during training. After training, the normalizing flow is easily recovered from the potential functions.

Paper Structure

This paper contains 19 sections, 7 theorems, 49 equations, 3 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. CNFs from entropic optimal transport
  4. Transport maps and normalizing flows
  5. Geodesics flows in the $2$-Wasserstein space $\mathbb{W}_2(\mathbb{R}^d)$
  6. Entropy-regularized $2$-Wasserstein distance
  7. A bridge between CNFs and potentials: the dynamic formulation
  8. Recovering the flow and density:
  9. Numerics
  10. Alternating between optimizing $\varphi$ and $\psi$
  11. Fast approximate $(c,\varepsilon)$-transform
  12. Constructing the CNF and the velocity field $v_t$
  13. Speeding optimization by reinforcing the $(c,\varepsilon)$-transform
  14. Examples
  15. Discussion and future work
  16. ...and 4 more sections

Key Result

Theorem 1

Let $\varepsilon>0$ be a positive number, $\Omega\subset\mathbb{R}^d$ be a compact set, $p_\mathcal{D},p_{\mathcal{N}} \in \mathcal{P}(\Omega)$. Then given $\varphi \in {{\rm L}^{\exp}_{\varepsilon}}(\mathbb{R}^d;{\rm d}x)$ and $\psi \in {{\rm L}^{\exp}_{\varepsilon}}(\mathbb{R}^d;{\rm d}y)$, the fo Moreover, the optimal coupling $\gamma^{\text{opt}}_{\varepsilon}$ is also the (unique) minimizer f

Figures (3)

  • Figure 1: Using the learned Entropy-Kantorovich potentials and \ref{['eq:\npunchline']}, the vector field $v_t$ (black arrows) recovered from the potentials creates a CNF between the checkerboard distribution (at $t=0$) and the standard Normal distribution (at $t=1$). Log-densities of the distributions along the flow are shown with the heat map.
  • Figure 1: Flow between two Normal distributions: the source distribution has a degenerate covariance structure and the target is the standard Normal distribution. (Top) $W_2$ geodesics (Bottom) The entropy-regularized interpolation.
  • Figure 2: Estimated densities and generated samples using Entropy-Kantorovic potentials, on 2D examples. (Top row) Ground-truth log-densities; (Middle row) Our approximated log-density $\rho_0^\varepsilon$; (Bottom row) Generated samples flowing from standard Normal.

Theorems & Definitions (10)

  • Theorem 1: Proposition 2.11, DMaGer19
  • Proposition 1: Lemma 2.6 in DMaGer19
  • Theorem 2: Brenier
  • Theorem 3
  • Definition 1
  • Theorem 4
  • Definition 2
  • Proposition 2: $(\mathcal{P}_p(\Omega),W_2)$ is a geodesic space
  • Proposition 3
  • Example 1: Comparing geodesics and entropic interpolation for Gaussian distributions