Discretized Gradient Flow for Manifold Learning in the Space of Embeddings
Dara Gold, Steven Rosenberg
TL;DR
This work addresses discretized gradient flow for manifold learning by performing optimization in the infinite-dimensional space ${\mathcal E}= {\rm Emb}(M,\mathbb{R}^N)$ of smooth embeddings. The authors show that, for a diffeomorphism-invariant penalty $P$, the gradient is pointwise normal to the embedded manifold, enabling updates along normal directions while preserving embeddings; they derive an explicit lower bound $t^* = \min\{K^{-1}, \delta/3\}$ on the step length in a fixed normal direction, with $K$ the maximal principal curvature and $\delta$ a reach-like parameter computable via a quantitative implicit-function theorem. Their main contribution is a constructive bound that quantifies how well discretized gradient flow approximates the smooth flow, bridging infinite-dimensional geometric analysis with practical optimization for manifold learning. This framework provides a principled route to implement discretized updates without reducing to finite-dimensional parameterizations, and it lays groundwork for computational strategies that maintain embedding validity during optimization.
Abstract
Gradient descent, or negative gradient flow, is a standard technique in optimization to find minima of functions. Many implementations of gradient descent rely on discretized versions, i.e., moving in the gradient direction for a set step size, recomputing the gradient, and continuing. In this paper, we present an approach to manifold learning where gradient descent takes place in the infinite dimensional space $\mathcal{E} = {\rm Emb}(M,\mathbb{R}^N)$ of smooth embeddings $φ$ of a manifold $M$ into $\mathbb{R}^N$. Implementing a discretized version of gradient descent for $P:\mathcal{E}\to {\mathbb R}$, a penalty function that scores an embedding $φ\in \mathcal{E}$, requires estimating how far we can move in a fixed direction -- the direction of one gradient step -- before leaving the space of smooth embeddings. Our main result is to give an explicit lower bound for this step length in terms of the Riemannian geometry of $φ(M)$. In particular, we consider the case when the gradient of $P$ is pointwise normal to the embedded manifold $φ(M)$. We prove this case arises when $P$ is invariant under diffeomorphisms of $M$, a natural condition in manifold learning.