Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Neural Ordinary Differential Equations on Manifolds

Luca Falorsi, Patrick Forré

TL;DR

This work extends Neural Ordinary Differential Equations to smooth manifolds to enable continuous, invertible transformations for normalizing flows on spaces with nontrivial topology. It introduces a manifold-centric framework using vector fields to generate diffeomorphisms via flows, and formalizes cotangent lifts as a geometric means to backpropagate through these flows with Hamiltonian structure. The Continuous Normalizing Flow on manifolds (MCNF) evolves densities along manifold flows using the divergence of the vector field, providing a practical recipe for density estimation on curved spaces. The approach promises scalable, topology-aware probabilistic modeling on manifolds, with demonstrations on hyperspheres and guidance for broader manifold settings and future improvements like Monte Carlo divergence and optimal transport regularization.

Abstract

Normalizing flows are a powerful technique for obtaining reparameterizable samples from complex multimodal distributions. Unfortunately current approaches fall short when the underlying space has a non trivial topology, and are only available for the most basic geometries. Recently normalizing flows in Euclidean space based on Neural ODEs show great promise, yet suffer the same limitations. Using ideas from differential geometry and geometric control theory, we describe how neural ODEs can be extended to smooth manifolds. We show how vector fields provide a general framework for parameterizing a flexible class of invertible mapping on these spaces and we illustrate how gradient based learning can be performed. As a result we define a general methodology for building normalizing flows on manifolds.

Neural Ordinary Differential Equations on Manifolds

TL;DR

This work extends Neural Ordinary Differential Equations to smooth manifolds to enable continuous, invertible transformations for normalizing flows on spaces with nontrivial topology. It introduces a manifold-centric framework using vector fields to generate diffeomorphisms via flows, and formalizes cotangent lifts as a geometric means to backpropagate through these flows with Hamiltonian structure. The Continuous Normalizing Flow on manifolds (MCNF) evolves densities along manifold flows using the divergence of the vector field, providing a practical recipe for density estimation on curved spaces. The approach promises scalable, topology-aware probabilistic modeling on manifolds, with demonstrations on hyperspheres and guidance for broader manifold settings and future improvements like Monte Carlo divergence and optimal transport regularization.

Abstract

Normalizing flows are a powerful technique for obtaining reparameterizable samples from complex multimodal distributions. Unfortunately current approaches fall short when the underlying space has a non trivial topology, and are only available for the most basic geometries. Recently normalizing flows in Euclidean space based on Neural ODEs show great promise, yet suffer the same limitations. Using ideas from differential geometry and geometric control theory, we describe how neural ODEs can be extended to smooth manifolds. We show how vector fields provide a general framework for parameterizing a flexible class of invertible mapping on these spaces and we illustrate how gradient based learning can be performed. As a result we define a general methodology for building normalizing flows on manifolds.

Paper Structure

This paper contains 16 sections, 7 theorems, 41 equations, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Vector fields and flows on manifolds
  3. Cotangent lift
  4. Cotangent lift in local coordinates
  5. Cotangent lift on embedded submanifolds
  6. Continuous normalizing flows on manifolds
  7. Related Work
  8. Conclusion and future work
  9. Proof of Concept Experiments
  10. Parameterizing vector fields on manifolds
  11. Local frames and global constraints
  12. Generators of vector fields
  13. Time dependent vector fields
  14. Embedded submanifolds of $\mathbb{R}^m$
  15. Embedded Riemannian Submanifolds
  16. ...and 1 more sections

Key Result

Theorem 1

Let $X$ a smooth time dependent vector field, then for any point $(t_0,p_0)\in \mathbb{R}\times M$ there exist a unique maximal integral curve $\gamma:t_0\in J\to M$ with starting point $q_0$, and starting time $t_0$ denoted by $\gamma(t;t_0,q_0)$. We call $\gamma$ a solution of the Cauchy problem: Moreover the map $(t_0, q_0) \to \gamma(t; t_0, q_0)$ is smooth on a neighborhood of $(t_0, q_0)$.

Figures (1)

  • Figure 1: Learned density on $S^2$

Theorems & Definitions (17)

  • Definition 1
  • Theorem 1: Theorem 2.15 in AgrBarBos17
  • Definition 2
  • Definition 3
  • Theorem 2: Continuity equation
  • Definition 4
  • Theorem 3
  • proof
  • Definition 5
  • Theorem 4: Whitney Embedding Theorem, 6.15 in lee2013smooth
  • ...and 7 more