Global Convergence of High-Order Regularization Methods with Sums-of-Squares Taylor Models
Wenqi Zhu, Coralia Cartis
TL;DR
This work develops an adaptive-regularization framework that embeds a Sum-of-Squares (SoS) Taylor model into high-order ($p\ge3$) optimization for nonconvex smooth functions. Each iteration minimizes a tractable SoS-based polynomial model via semidefinite programming, with a data-dependent perturbation $\delta=\epsilon^{a}$ ensuring well-conditioned subproblems. The authors prove global convergence and establish worst-case evaluation complexities: $\mathcal{O}(\epsilon^{-2})$ in general nonconvex settings, and improved rates $\mathcal{O}(\epsilon^{-1/p})$ for odd $p$ or $\mathcal{O}(\epsilon^{-(p+1)/p})$ for even $p$ when $f$ is strongly convex. The results rely on uniform upper bounds for regularization parameters and detailed analysis of function-value decrease in successful iterations. This work provides the first global-rate analysis for a tractable high-order subproblem in nonconvex optimization and sets the stage for further refinements and practical SDP-based implementations.
Abstract
High-order tensor methods that employ Taylor-based local models (of degree $p\ge 3$) within adaptive regularization frameworks have been recently proposed for both convex and nonconvex optimization problems. They have been shown to have superior, and even optimal, worst-case global convergence rates and local rates compared to Newton's method. Finding rigorous and efficient techniques for minimizing the Taylor polynomial sub-problems remains a challenging aspect for these algorithms. Ahmadi et al. recently introduced a tensor method based on sum-of-squares (SoS) reformulations, so that each Taylor polynomial sub-problem in their approach can be tractably minimized using semidefinite programming (SDP); however, the global convergence and complexity of their method have not been addressed for general nonconvex problems. This paper introduces an algorithmic framework that combines the Sum of Squares (SoS) Taylor model with adaptive regularization techniques for nonconvex smooth optimization problems. Each iteration minimizes an SoS Taylor model, offering a polynomial cost per iteration. For general nonconvex functions, the worst-case evaluation complexity bound is $\mathcal{O}(ε^{-2})$, while for strongly convex functions, an improved evaluation complexity bound of $\mathcal{O}(ε^{-\frac{1}{p}})$ is established. To the best of our knowledge, this is the first global rate analysis for an adaptive regularization algorithm with a tractable high-order sub-problem in nonconvex smooth optimization, opening the way for further improvements.