A Non-Monotone Line-Search Method for Minimizing Functions with Spurious Local Minima
Zohreh Aminifard, Geovani Nunes Grapiglia
TL;DR
Problem: minimize a differentiable function $f:\mathbb{R}^n \to \mathbb{R}$ with many spurious local minima. Approach: a non-monotone line-search based on a relaxed Armijo condition $f(x_k^+) \le f(x_k) + \rho \alpha_k \langle \nabla f(x_k), d_k \rangle + \nu_k$, with a Metropolis-inspired relaxation term $\nu_k$ designed to allow controlled increases and help escape local basins. Contributions: (i) Algorithm 1 with a dimensionally consistent relaxation $\nu_k = \sigma \exp(-\max\{\theta, (f_{l(k)}-f(x_k^+))/(\rho \alpha_k \langle \nabla f(x_k), d_k \rangle)\}) \ln(k+1)$; (ii) explicit worst-case complexity bounds: $T(\epsilon) = O(\epsilon^{-2})$ for $\theta>1$ and $T(\epsilon) = O(\epsilon^{-2/\theta})$ for $\theta\in(0,1)$; and (iii) numerical results showing improved performance over existing non-monotone schemes. Significance: the method provides tunable exploration via the parameter $\theta$, improving robustness to spurious local minima and offering practical performance gains.
Abstract
In this paper, we propose a new non-monotone line-search method for smooth unconstrained optimization problems with objective functions that have many non-global local minimizers. The method is based on a relaxed Armijo condition that allows a controllable increase in the objective function between consecutive iterations. This property helps the iterates escape from nearby local minimizers in the early iterations. For objective functions with Lipschitz continuous gradients, we derive worst-case complexity estimates on the number of iterations needed for the method to find approximate stationary points. Numerical results are presented, showing that the new method can significantly outperform other non-monotone methods on functions with spurious local minima.