Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Adaptive Complexity of Finding a Stationary Point

Huanjian Zhou, Andi Han, Akiko Takeda, Masashi Sugiyama

TL;DR

The paper establishes tight adaptive-complexity characterizations for finding $oldsymbol{ abla f(oldsymbol{x})}$-stationary points in non-convex optimization. It shows that in high dimensions ($d = ilde{ Omega}(oldsymbol{oldsymbol{oldsymbol{oldsymbol{oldsymbol{}}}}})$) parallelization provides no acceleration: any algorithm requires at least $oldsymbol{ ilde{oldsymbol{oldsymbol{oldsymbol{oldsymbol{}}}}}} ig( rac{1}{oldsymbol{oldsymbol{oldsymbol{oldsymbol{oldsymbol{}}}}}ig)^{(1+p)/p}$ rounds to reach $oldsymbol{ abla f} ext{-stationarity}$, even with polynomially many queries per round, matching one-query-per-round baselines such as gradient descent, cubic-regularized Newton, and ARp. In constant dimensions, the authors introduce Gradient Flow Grid Trapping (GFGT), bridging grid search and GFPT, and prove constant-round solvability with per-round queries scaling towards $ ilde{O}( rac{1}{oldsymbol{oldsymbol{oldsymbol{oldsymbol{oldsymbol{}}}}}^{(d-1)/2})}$, while also providing matching lower bounds for $ ilde{O}( ext{log}(1/oldsymbol{oldsymbol{oldsymbol{oldsymbol{oldsymbol{}}}}))$ rounds. The results, including novel random-partition constructions and boundary-unreachability techniques, imply that adaptive parallelization does not fundamentally speed up non-convex optimization in these regimes. The work also connects to stochastic settings and highlights adaptive optimality of classical methods under Lipschitz $p$-th order smoothness. Overall, the findings resolve open questions about the power of adaptivity in non-convex optimization and delineate the regimes where parallelization can be costly or ineffective.

Abstract

In large-scale applications, such as machine learning, it is desirable to design non-convex optimization algorithms with a high degree of parallelization. In this work, we study the adaptive complexity of finding a stationary point, which is the minimal number of sequential rounds required to achieve stationarity given polynomially many queries executed in parallel at each round. For the high-dimensional case, i.e., $d = \widetildeΩ(\varepsilon^{-(2 + 2p)/p})$, we show that for any (potentially randomized) algorithm, there exists a function with Lipschitz $p$-th order derivatives such that the algorithm requires at least $\varepsilon^{-(p+1)/p}$ iterations to find an $\varepsilon$-stationary point. Our lower bounds are tight and show that even with $\mathrm{poly}(d)$ queries per iteration, no algorithm has better convergence rate than those achievable with one-query-per-round algorithms. In other words, gradient descent, the cubic-regularized Newton's method, and the $p$-th order adaptive regularization method are adaptively optimal. Our proof relies upon novel analysis with the characterization of the output for the hardness potentials based on a chain-like structure with random partition. For the constant-dimensional case, i.e., $d = Θ(1)$, we propose an algorithm that bridges grid search and gradient flow trapping, finding an approximate stationary point in constant iterations. Its asymptotic tightness is verified by a new lower bound on the required queries per iteration. We show there exists a smooth function such that any algorithm running with $Θ(\log (1/\varepsilon))$ rounds requires at least $\widetildeΩ((1/\varepsilon)^{(d-1)/2})$ queries per round. This lower bound is tight up to a logarithmic factor, and implies that the gradient flow trapping is adaptively optimal.

The Adaptive Complexity of Finding a Stationary Point

TL;DR

The paper establishes tight adaptive-complexity characterizations for finding -stationary points in non-convex optimization. It shows that in high dimensions () parallelization provides no acceleration: any algorithm requires at least rounds to reach , even with polynomially many queries per round, matching one-query-per-round baselines such as gradient descent, cubic-regularized Newton, and ARp. In constant dimensions, the authors introduce Gradient Flow Grid Trapping (GFGT), bridging grid search and GFPT, and prove constant-round solvability with per-round queries scaling towards , while also providing matching lower bounds for rounds. The results, including novel random-partition constructions and boundary-unreachability techniques, imply that adaptive parallelization does not fundamentally speed up non-convex optimization in these regimes. The work also connects to stochastic settings and highlights adaptive optimality of classical methods under Lipschitz -th order smoothness. Overall, the findings resolve open questions about the power of adaptivity in non-convex optimization and delineate the regimes where parallelization can be costly or ineffective.

Abstract

In large-scale applications, such as machine learning, it is desirable to design non-convex optimization algorithms with a high degree of parallelization. In this work, we study the adaptive complexity of finding a stationary point, which is the minimal number of sequential rounds required to achieve stationarity given polynomially many queries executed in parallel at each round. For the high-dimensional case, i.e., , we show that for any (potentially randomized) algorithm, there exists a function with Lipschitz -th order derivatives such that the algorithm requires at least iterations to find an -stationary point. Our lower bounds are tight and show that even with queries per iteration, no algorithm has better convergence rate than those achievable with one-query-per-round algorithms. In other words, gradient descent, the cubic-regularized Newton's method, and the -th order adaptive regularization method are adaptively optimal. Our proof relies upon novel analysis with the characterization of the output for the hardness potentials based on a chain-like structure with random partition. For the constant-dimensional case, i.e., , we propose an algorithm that bridges grid search and gradient flow trapping, finding an approximate stationary point in constant iterations. Its asymptotic tightness is verified by a new lower bound on the required queries per iteration. We show there exists a smooth function such that any algorithm running with rounds requires at least queries per round. This lower bound is tight up to a logarithmic factor, and implies that the gradient flow trapping is adaptively optimal.
Paper Structure (48 sections, 27 theorems, 71 equations, 2 figures, 1 table)

This paper contains 48 sections, 27 theorems, 71 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Our results
  3. Lower bounds in high dimension.
  4. Lower and upper bounds in constant dimension.
  5. Technical overview
  6. High-dimensional case
  7. Prior arguement fails under polynomial queries.
  8. Our construction by random partition.
  9. Comparison with construction for stochastic settings.
  10. Trap the flow within a constant number of iterations in constant-dimensional space
  11. Boundary unreachability implies stationary point inside.
  12. Our method: Gradient Flow Grid Trapping (GFGT).
  13. Boundary unreachability during our algorithm.
  14. Comparison with constant-iteration algorithms for local minimum search on grid graphs.
  15. Constant-dimensional lower bound.
  16. ...and 33 more sections

Key Result

Theorem 1

For $d= \widetilde{\Omega}\left(\varepsilon^{-(2+2p)/{p}}\right)$ and $p$-th order smooth function, any randomized optimizer needs $\Omega(\varepsilon^{-(1+p)/p})$ sequential rounds to find $\varepsilon$-stationary points even with $\mathsf{poly}(d)$ queries per round.

Figures (2)

  • Figure 1: Illustration of iterate update on 2-dimensional space. We plot trap barriers as black grids and queried points on the trap barrier as blue dots. (a) Current iterate location. (b) If all the queried points are $\varepsilon_t$-unreachable from $\boldsymbol x_t$, then $\boldsymbol x^{t+1} = \boldsymbol x^t$. (c) If some queried points are $\varepsilon_t$-reachable, we select $\boldsymbol{x}^{t+1}$ as the point with smallest function value.
  • Figure 2: Illustration of domain compression on 2-dimensional space depending on the location of $\boldsymbol{x}^{t+1}$. The shaded areas represent the compressed domain for the next iteration. (a,b) If $\boldsymbol{x}^{t+1}$ is close to a boundary (in this case, the left boundary), the domain is extended only to the other directions (top, bottom, right). (c,d) If $\boldsymbol{x}^{t+1}$ is not close to any boundary, the domain is extended for to directions, depending on whether $\boldsymbol{x}^{t+1}$ is on the trap barrier.

Theorems & Definitions (31)

  • Theorem 1: informal, see Theorem \ref{['the:main']}
  • Theorem 2: informal, see Theorem \ref{['the:main3']}
  • Theorem 3: informal, see Theorem \ref{['the:main2']}
  • Theorem 4
  • Lemma 5: carmon2020lower
  • Lemma 6: Smoothness and boundness of $g_\mathcal{P}$
  • Lemma 7: Smoothness and boundness of $f_\mathcal{P}$
  • Lemma 8: Characterization of output
  • Lemma 9: Small weighted partition implies large gradient norm
  • Theorem 10
  • ...and 21 more