A Novel First-order Method with Event-driven Objective Evaluations
Christian Varner, Vivak Patel
TL;DR
This work tackles optimization where evaluating the objective $F(\theta)$ is costly (due to numerical integration) while gradient evaluations are cheap, a pattern common in semi-parametric statistics and control. It identifies anti-convergence risks for objective-function-free optimization (OFFO) methods on non-convex, locally Lipschitz smooth objectives and proposes a novel gradient method with event-driven objective evaluations, a non-sequential Armijo-type acceptance, and an adaptive step-size strategy that leverage triggering events for economical, reliable progress. The main contributions are (i) formal anti-convergence results and a pathological construction to justify the need for objective checks, (ii) a concrete algorithm that triggers objective evaluations only when necessary, (iii) a global convergence analysis and convergence-rate results under general smoothness assumptions, and (iv) numerical experiments on quasi-likelihood problems demonstrating improved reliability and reduced objective evaluations compared to OFFO methods. Overall, the proposed event-driven approach offers a practical, theoretically grounded solution for expensive-objective optimization problems with cheap gradients, applicable to semi-parametric statistics and related global optimization tasks.
Abstract
Arising in semi-parametric statistics, control applications, and as sub-problems in global optimization methods, certain optimization problems can have objective functions requiring numerical integration to evaluate, yet gradient function evaluations that are relatively cheap. For such problems, typical optimization methods that require multiple evaluations of the objective for each new iterate become computationally expensive. In light of this, optimization methods that avoid objective function evaluations are attractive, yet we show anti-convergence behavior for these methods on the problem class of interest. To address this gap, we develop a novel gradient algorithm that only evaluates the objective function when specific events are triggered and propose a step size scheme that greedily takes advantage of the properties induced by these triggering events. We prove that our methodology has global convergence guarantees under the most general smoothness conditions, and show through extensive numerical results that our method performs favorably on optimization problems arising in semi-parametric statistics.