Some iterative algorithms on Riemannian manifolds and Banach spaces with good global convergence guarantee
Tuyen Trung Truong
TL;DR
This work develops iterative optimization algorithms for both finite- and infinite-dimensional settings—Riemannian manifolds and Banach spaces—with strong global convergence guarantees to local minima and robust saddle-point avoidance. It introduces Backtracking GD variants, Local Backtracking GD, and Backtracking New Q-Newton's methods, extending them to manifolds via strong local retractions and to Banach spaces via generalized shyness and weak convergence, including stable-center manifold analyses. Theoretical guarantees ensure that cluster points are critical points, sequences converge to local minima under KL or finite critical-point conditions, and generic randomness prevents convergence to saddles; in Banach spaces, the results are framed in terms of weak convergence and shy sets, with quadratic convergence near non-degenerate minima for BNQN. The framework is demonstrated on problems such as sphere optimization and variational PDE contexts, highlighting broad applicability to constrained optimization and infinite-dimensional variational problems.
Abstract
In this paper, we introduce some new iterative optimisation algorithms on Riemannian manifolds and Hilbert spaces which have good global convergence guarantees to local minima. More precisely, these algorithms have the following properties: If $\{x_n\}$ is a sequence constructed by one such algorithm then: - Finding critical points: Any cluster point of $\{x_n\}$ is a critical point of the cost function $f$. - Convergence guarantee: Under suitable assumptions, the sequence $\{x_n\}$ either converges to a point $x^*$, or diverges to $\infty$. - Avoidance of saddle points: If $x_0$ is randomly chosen, then the sequence $\{x_n\}$ cannot converge to a saddle point. Our results apply for quite general situations: the cost function $f$ is assumed to be only $C^2$ or $C^3$, and either $f$ has at most countably many critical points (which is a generic situation) or satisfies certain Lojasiewicz gradient inequalities. To illustrate the results, we provide a nice application with optimisation over the unit sphere in a Euclidean space. As for tools needed for the results, in the Riemannian manifold case we introduce a notion of "strong local retraction" and (to deal with Newton's method type) a notion of "real analytic-like strong local retraction". In the case of Banach spaces, we introduce a slight generalisation of the notion of "shyness", and design a new variant of Backtracking New Q-Newton's method which is more suitable to the infinite dimensional setting (and in the Euclidean setting is simpler than the current versions).