Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Balancing Gradient and Hessian Queries in Non-Convex Optimization

Deeksha Adil, Brian Bullins, Aaron Sidford, Chenyi Zhang

TL;DR

This paper studies the computation of ε-critical points for non-convex functions by balancing gradient and Hessian queries under L_1 Lipschitz gradients and L_2 Lipschitz Hessians. It introduces a δ-approximate Hessian oracle and builds a framework around accelerated descent in a Hessian-induced norm, plus restarts and negative-curvature steps to guarantee convergence. The main contribution is a trade-off theorem that yields ε-critical points with a tunable number of Hessian queries n_H and gradient queries bounded by polylogarithmic factors, recovering known gradient-only rates as special cases and improving dimension-dependent and single-Hessian scenarios. The results advance the theoretical understanding of optimal query complexities in non-convex optimization, with potential implications for large-scale problems where Hessian computations are costly but gradient evaluations are cheaper. All bounds are presented with careful handling of δ, L_1, L_2, Δ, and ε, and the work includes a reduction to remove the L_1-gradient assumption while preserving the core gradient-Hessian trade-offs.

Abstract

We develop optimization methods which offer new trade-offs between the number of gradient and Hessian computations needed to compute the critical point of a non-convex function. We provide a method that for any twice-differentiable $f\colon \mathbb R^d \rightarrow \mathbb R$ with $L_2$-Lipschitz Hessian, input initial point with $Δ$-bounded sub-optimality, and sufficiently small $ε> 0$, outputs an $ε$-critical point, i.e., a point $x$ such that $\|\nabla f(x)\| \leq ε$, using $\tilde{O}(L_2^{1/4} n_H^{-1/2}Δε^{-9/4})$ queries to a gradient oracle and $n_H$ queries to a Hessian oracle for any positive integer $n_H$. As a consequence, we obtain an improved gradient query complexity of $\tilde{O}(d^{1/3}L_2^{1/2}Δε^{-3/2})$ in the case of bounded dimension and of $\tilde{O}(L_2^{3/4}Δ^{3/2}ε^{-9/4})$ in the case where we are allowed only a \emph{single} Hessian query. We obtain these results through a more general algorithm which can handle approximate Hessian computations and recovers the state-of-the-art bound of computing an $ε$-critical point with $O(L_1^{1/2}L_2^{1/4}Δε^{-7/4})$ gradient queries provided that $f$ also has an $L_1$-Lipschitz gradient.

Balancing Gradient and Hessian Queries in Non-Convex Optimization

TL;DR

This paper studies the computation of ε-critical points for non-convex functions by balancing gradient and Hessian queries under L_1 Lipschitz gradients and L_2 Lipschitz Hessians. It introduces a δ-approximate Hessian oracle and builds a framework around accelerated descent in a Hessian-induced norm, plus restarts and negative-curvature steps to guarantee convergence. The main contribution is a trade-off theorem that yields ε-critical points with a tunable number of Hessian queries n_H and gradient queries bounded by polylogarithmic factors, recovering known gradient-only rates as special cases and improving dimension-dependent and single-Hessian scenarios. The results advance the theoretical understanding of optimal query complexities in non-convex optimization, with potential implications for large-scale problems where Hessian computations are costly but gradient evaluations are cheaper. All bounds are presented with careful handling of δ, L_1, L_2, Δ, and ε, and the work includes a reduction to remove the L_1-gradient assumption while preserving the core gradient-Hessian trade-offs.

Abstract

We develop optimization methods which offer new trade-offs between the number of gradient and Hessian computations needed to compute the critical point of a non-convex function. We provide a method that for any twice-differentiable with -Lipschitz Hessian, input initial point with -bounded sub-optimality, and sufficiently small , outputs an -critical point, i.e., a point such that , using queries to a gradient oracle and queries to a Hessian oracle for any positive integer . As a consequence, we obtain an improved gradient query complexity of in the case of bounded dimension and of in the case where we are allowed only a \emph{single} Hessian query. We obtain these results through a more general algorithm which can handle approximate Hessian computations and recovers the state-of-the-art bound of computing an -critical point with gradient queries provided that also has an -Lipschitz gradient.
Paper Structure (37 sections, 51 theorems, 326 equations, 1 table)

This paper contains 37 sections, 51 theorems, 326 equations, 1 table.

Table of Contents

  1. Introduction
  2. Our Results
  3. Generalizing Prior Gradient Methods.
  4. Gradient-Hessian Trade-offs for Functions with Unbounded Smoothness.
  5. Dimension-Dependent Critical Point Computation.
  6. Additional Related Work.
  7. Our Algorithms
  8. Critical or Progress using Approximate Hessians
  9. Restarted Approximate Hessian AGD
  10. Removing the $L_1$-Lipschitz Gradient Assumption
  11. Proof of Theorem \ref{['thm:main']}
  12. Conclusion
  13. Overview and Notation of the Appendix
  14. Notation.
  15. Comparison Between jiang2024improved and Prior Works
  16. ...and 22 more sections

Key Result

Theorem 1

Let $f \colon \mathbb{R}^d \mapsto \mathbb{R}$ have $L_1$-Lipschitz gradient and $L_2$-Lipschitz Hessian. There is an algorithm which given any $x^{(0)} \in \mathbb{R}^d$ with $\Delta$-bounded sub-optimality with respect to $f$, positive integer $n_H$, and $0 < \epsilon\leq \min\{L_1^2L_2^{-1},\Delt queries to a gradient oracle for $f$, where

Theorems & Definitions (97)

  • Definition 1: Approximate Hessian Oracle
  • Theorem 1: Main Result
  • Corollary 1
  • Theorem 2
  • Theorem 3
  • Lemma 1
  • proof : Proof of Theorem \ref{['thm:Restarted-Hessian-AGD-w-L1']}
  • Theorem 4
  • Corollary 2
  • proof
  • ...and 87 more