A Derivative-Free Saddle-search Algorithm With Linear Convergence Rate

TL;DR

The paper tackles the problem of locating saddle points in high-dimensional energy landscapes when derivatives are unavailable by designing a nested, derivative-free saddle-search algorithm. The approach combines an inner zeroth-order eigenvector search with an outer zeroth-order saddle-search, using Gaussian-smoothed estimators to replace gradients and Hessians, and employing a decaying or constant step-size regime. The authors establish almost-sure convergence of the inner step, prove outer-step convergence under decaying parameters, and demonstrate linear convergence with a constant step size to a neighborhood whose radius scales with the smoothing length, supported by comprehensive numerical experiments on benchmark potentials, implicit objectives, high-dimensional Rosenbrock problems, and neural network loss landscapes. The work offers a practical, derivative-free path to transition-state computation in contexts where derivative information is inaccessible, with potential extensions including momentum, directional smoothing, and variance-reduction techniques.

Abstract

We propose a derivative-free saddle-search algorithm designed to locate transition states using only function evaluations. The algorithm employs a nested architecture consisting of an inner eigenvector search and an outer saddle-point search. Through rigorous numerical analysis, we prove the almost sure convergence of the inner step under suitable assumptions. Furthermore, we establish the convergence of the outer search using a decaying step size, while demonstrating linear convergence under constant step size and boundedness conditions. Numerical experiments are provided to validate our theoretical results and demonstrate the algorithm's practical applicability.

Figures (6)

• Figure 1: The standard deviations are computed over 10000 samples of $\mathbf{H} \mathbf{v}$ (Hessian estimator) and $H_\mathbf{v}$ (Hessian-vector estimator), where $\mathbf{v}$ is a fixed vector drawn from $\mathcal{N}(\mathbf{0},\mathbf{I})$.
• Figure 1: Contour plot of MB potential and iteration points (a) of one-time run of \ref{['algorithm']}. Error plots of the iteration points with (b) $l=0.001, \alpha_{\mathbf{x}}\equiv 0.0001,n_\mathbf{x}^{max}=1000, n_\mathbf{v}^{max}=100$, $\alpha_\mathbf{v}\equiv0.0002$ and (c) $l=0.0001, \alpha_\mathbf{x}\equiv 0.0001,n_\mathbf{x}^{max}=1000, n_\mathbf{v}^{max}=100$, $\alpha_\mathbf{v}\equiv0.0002/d$.
• Figure 2: (a) Contour plot of \ref{['eq: implicit function']} and iteration points of one-time run of \ref{['algorithm']}. (b) Error plot with $l=0.1, \alpha_\mathbf{x}\equiv 0.01,n_\mathbf{x}^{max}=5000, n_\mathbf{v}^{max}=100$, $\alpha_\mathbf{v}\equiv0.0002/d$.
• Figure 3: (a) Energy landscape of the 2D Rosenbrock function and iteration points of one-time run of \ref{['algorithm']}. (b) Error plot of the iteration points of \ref{['algorithm']} with $l=0.0001, \alpha_\mathbf{x}\equiv 0.00001,n_\mathbf{x}^{max}=10000, n_\mathbf{v}^{max}=100$, $\alpha_\mathbf{v}\equiv0.0002/d$.
• Figure 4: (a)The error plots of the inner eigenvector search algorithm (\ref{['algorithm vector']}) with $d=2$ and $d=100$, using the Hessian approximation ($\mathbf{H}\mathbf{v}$) and the Hessian–vector approximation ($H_{\mathbf{v}}$), respectively. The step size is choosen as $\alpha(n) = \frac{1}{d(10n+10000)}$. (b) Error plots of \ref{['algorithm']} and the deterministic saddle-search algorithm in \ref{['eq: saddle algorithm']} applied to the modified Rosenbrock function with $d=1000,$$l=0.0001, \alpha_\mathbf{x}\equiv 0.000001,n_\mathbf{x}^{max}=100000, n_\mathbf{v}^{max}=100$, $\alpha_\mathbf{v}\equiv0.0002/d$.

Theorems & Definitions (41)

• Remark 3.1
• Remark 3.2
• Lemma 3.3: Almost unbiased estimator
• Proof 1
• Lemma 3.4
• Proof 2
• Lemma 3.5: Davis-Kahan theorem yu2015useful
• Lemma 3.6
• Lemma 4.1
• Proof 3