Convergence of high-index saddle dynamics for degenerate saddle points on critical manifolds

TL;DR

This work extends high-index saddle dynamics (HiSD) to degenerate saddle points situated on critical manifolds by leveraging Morse–Bott theory. It proves local asymptotic stability of the saddle manifold under the continuous HiSD flow and establishes linear convergence for discrete HiSD when the index satisfies $k \\in \\{s,\\dots, s+m\\}$, even in the presence of zero Hessian eigenvalues. A gradient-alignment mechanism is derived, showing the gradient increasingly aligns with a Hessian eigenvector corresponding to the smallest nonzero eigenvalue, while maintaining robustness to variations in the unstable directions. Numerical experiments on neural-network loss landscapes demonstrate rapid convergence of momentum-accelerated HiSD variants to degenerate saddles and validate the theoretical findings, with Nesterov acceleration offering the fastest convergence in the tested scenarios.

Abstract

The high-index saddle dynamics (HiSD) method provides a powerful framework for finding saddle points and constructing solution landscapes. While originally derived for nondegenerate critical points, HiSD has demonstrated empirical success in degenerate cases, where the Hessian matrix exhibits zero eigenvalues. However, the mathematical and numerical analysis of HiSD for degenerate saddle points remains unexplored. In this paper, utilizing Morse-Bott functions, we present a rigorous analysis of HiSD for computing degenerate saddle points on a critical manifold. We prove the local convergence of the continuous HiSD and establish the linear convergence rate of the discrete HiSD algorithm. Furthermore, we provide a theoretical explanation for the gradient alignment tendency, revealing that the gradient direction asymptotically aligns with a specific Hessian eigenvector. Our analysis also elucidates the flexibility in selecting the index for HiSD in the context of degenerate saddle points. We validate our analytical results through numerical experiments on neural-network loss landscapes and demonstrate that momentum-accelerated variants of HiSD achieve rapid convergence to degenerate saddle points.

Figures (2)

• Figure 1: Plots of (a) gradient norm $\|\nabla E(\theta^{(t)})\|_2$, (b) distance to the saddle manifold $\|\theta^{(t)}-\hat{\theta}^{(t)}\|_2$ and (c) extent of gradient alignment ${z_t}/{\|\nabla E(\theta^{(t)})\|_2}$ with respect to the iteration number $t$ for HiSD with different indices $k$.
• Figure 2: Plots of (a) gradient norm $\|\nabla E(\theta^{(t)})\|_2$ and (b) distance to the saddle manifold $\|\theta^{(t)}-\hat{\theta}^{(t)}\|_2$ with respect to the iteration number $t$ for different acceleration methods.

Theorems & Definitions (29)

• Definition 2.1: Morse function
• Definition 2.2: nondegenerate critical submanifold
• Definition 2.3: Morse--Bott function
• Lemma 2.4: Morse--Bott lemma
• Remark 3.1
• Remark 3.2
• Remark 3.4
• Lemma 3.5: distance between subspaces
• Lemma 3.6
• Lemma 3.7: order‐separated gradient decomposition