Global Optimality Characterizations and Algorithms for Minimizing Quartically-Regularized Third-Order Taylor Polynomials

Abstract

High-order methods for convex and nonconvex optimization, particularly $p$th-order Adaptive Regularization Methods (AR$p$), have attracted significant research interest by naturally incorporating high-order Taylor models into adaptive regularization frameworks, resulting in algorithms with faster global and local convergence rates than first- and second-order methods. This paper establishes global optimality conditions for general, nonconvex cubic polynomials with quartic regularization. These criteria generalise existing results, recovering the optimality results for regularized quadratic polynomials, and can be further simplified in the low-rank and diagonal tensor cases. Under suitable assumptions on the Taylor polynomial, we derive a lower bound for the regularization parameter such that the necessary and sufficient criteria coincide, establishing a connection between this bound and the subproblem's convexification and sum-of-squares (SoS) convexification techniques. Leveraging the optimality characterization, we develop a Diagonal Tensor Method (DTM) for minimizing quartically-regularized cubic Taylor polynomials by iteratively minimizing a sequence of local models that incorporate both diagonal cubic terms and quartic regularization (DTM model). We show that the DTM algorithm is provably convergent, with a global evaluation complexity of $\mathcal{O}(ε^{-3/2})$. Furthermore, when special structure is present (such as low rank or diagonal), DTM can exactly solve the given problem (in one iteration). In our numerical experiments, we propose practical DTM variants that exploit local problem information for model construction, which we then show to be competitive with cubic regularization and other subproblem solvers, with superior performance on problems with special structure.

Figures (4)

• Figure 1: Diagonal Tensor Test Set: Parameter configurations are as follows. Standard Set: a = 10, b = 20, c = 20, $\sigma = 100$. Ill-Conditioned Hessian Set: a = 10, c = 20, $H$ is a diagonal matrix uniformly distributed in $[10^{-6}, 10^3]$, $\sigma = 100$. Ill-Conditioned Tensor Set: a = 10, b = 20, $t$ has entries uniformly distributed in $[10^{-6}, 10^3]$, $\sigma = 500$.
• Figure 2: Low-Rank Tensor Test Set: Parameter configurations are as follows. a = 10, b = 20, c = 20, $\sigma = 100$, with rank $P = 1$ and $P = 4$, respectively.
• Figure 3: Global optimality condition and size of $\|T\|$ or $\sigma$ using the Diagonal Tensor Test Set: First plot: a = 10, b = 20, $\sigma = 100$, c$\in [5, 80]$. Second plot: a = 10, b = 20, c= 50, $\sigma$$\in [50, 300]$.
• Figure 4: CPU time comparison between the full tensor variant and the tensor-free variant. The test functions and setup are identical to those in \ref{['table:results']}.

Theorems & Definitions (62)

• Lemma 2.1
• Theorem 2.1
• proof
• Remark 2.1
• Theorem 2.2
• proof
• Remark 2.2
• Lemma 2.2
• proof
• Lemma 2.3