Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Nonmonotone higher-order Taylor approximation methods for composite problems

Yassine Nabou

TL;DR

The paper addresses composite optimization with a $p$-times differentiable smooth part and a convex non-smooth term, and proposes NHOTA, a nonmonotone higher-order Taylor method that avoids global Lipschitz and monotone descent assumptions. It proves global convergence to stationary points for nonconvex problems with rate $\mathcal{O}(k^{-\frac{p}{p+1}})$, and improves rates under the KL property; in the convex case it yields sublinear rates $\mathcal{O}(k^{-p})$. The method builds a regularized $p$-th order model and allows nonmonotone progress by updating a reference value $\mathcal{R}_k$, with descent guaranteed in aggregate. Numerical experiments on nonconvex phase retrieval illustrate competitive performance and robustness to nonmonotone steps. The framework broadens applicability by relaxing global smoothness constraints and opening paths to accelerated or constrained variants.

Abstract

We study composite optimization problems in which the smooth part of the objective function is \( p \)-times continuously differentiable, where \( p \geq 1 \) is an integer. Higher-order methods are known to be effective for solving such problems, as they speed up convergence rates. These methods often require, or implicitly ensure, a monotonic decrease in the objective function across iterations. Maintaining this monotonicity typically requires that the \( p \)-th derivative of the smooth part of the objective function is globally Lipschitz or that the generated iterates remain bounded. In this paper, we propose nonmonotone higher-order Taylor approximation (NHOTA) method for composite problems. Our method achieves the same nice global and rate of convergence properties as traditional higher-order methods while eliminating the need for global Lipschitz continuity assumptions, strict descent condition, or explicit boundedness of the iterates. Specifically, for nonconvex composite problems, we derive global convergence rate to a stationary point of order \( \mathcal{O}(k^{-\frac{p}{p+1}}) \), where \( k \) is the iteration counter. Moreover, when the objective function satisfies the Kurdyka-Łojasiewicz (KL) property, we obtain improved rates that depend on the KL parameter. Furthermore, for convex composite problems, our method achieves sublinear convergence rate of order \( \mathcal{O}(k^{-p}) \) in function values. Finally, preliminary numerical experiments on nonconvex phase retrieval problems highlight the promising performance of the proposed approach.

Nonmonotone higher-order Taylor approximation methods for composite problems

TL;DR

The paper addresses composite optimization with a -times differentiable smooth part and a convex non-smooth term, and proposes NHOTA, a nonmonotone higher-order Taylor method that avoids global Lipschitz and monotone descent assumptions. It proves global convergence to stationary points for nonconvex problems with rate , and improves rates under the KL property; in the convex case it yields sublinear rates . The method builds a regularized -th order model and allows nonmonotone progress by updating a reference value , with descent guaranteed in aggregate. Numerical experiments on nonconvex phase retrieval illustrate competitive performance and robustness to nonmonotone steps. The framework broadens applicability by relaxing global smoothness constraints and opening paths to accelerated or constrained variants.

Abstract

We study composite optimization problems in which the smooth part of the objective function is -times continuously differentiable, where is an integer. Higher-order methods are known to be effective for solving such problems, as they speed up convergence rates. These methods often require, or implicitly ensure, a monotonic decrease in the objective function across iterations. Maintaining this monotonicity typically requires that the -th derivative of the smooth part of the objective function is globally Lipschitz or that the generated iterates remain bounded. In this paper, we propose nonmonotone higher-order Taylor approximation (NHOTA) method for composite problems. Our method achieves the same nice global and rate of convergence properties as traditional higher-order methods while eliminating the need for global Lipschitz continuity assumptions, strict descent condition, or explicit boundedness of the iterates. Specifically, for nonconvex composite problems, we derive global convergence rate to a stationary point of order \( \mathcal{O}(k^{-\frac{p}{p+1}}) \), where is the iteration counter. Moreover, when the objective function satisfies the Kurdyka-Łojasiewicz (KL) property, we obtain improved rates that depend on the KL parameter. Furthermore, for convex composite problems, our method achieves sublinear convergence rate of order \( \mathcal{O}(k^{-p}) \) in function values. Finally, preliminary numerical experiments on nonconvex phase retrieval problems highlight the promising performance of the proposed approach.

Paper Structure

This paper contains 9 sections, 5 theorems, 51 equations, 1 figure, 1 algorithm.

Table of Contents

  1. Introduction and motivation
  2. Notations and preliminaries
  3. Nonmonotone higher-order Tensor method
  4. Nonconvex convergence analysis
  5. Improved convergence rates under KL
  6. Convex convergence analysis
  7. Numerical illustrations
  8. Nonconvex phase retrieval
  9. Conclusion

Key Result

lemma thmcounterlemma

Let Assumption ass:1 be satisfied and let $(x_k)_{k\geq 0}$ be generated by NHOTA alg:NOHTA. Then, the following statements hold for all $k\geq 0$:

Figures (1)

  • Figure 1: Performance of NHOTA in terms of the gradient norm $\|\nabla F(x_k)\|$, objective value $f(x_k)$, and CPU time across iterations. The term $\|\nabla F(x_k) + \lambda g_k\| \approx \|\nabla F(x_k)\|$ for all $g \in \partial h(x_k)$ is omitted due to $\lambda = 10^{-5}$ and $\|g_k\| \leq 10$, as its contribution is negligible.

Theorems & Definitions (15)

  • definition thmcounterdefinition
  • lemma thmcounterlemma
  • proof
  • remark thmcounterremark
  • theorem 1
  • proof
  • remark thmcounterremark
  • lemma thmcounterlemma
  • proof
  • theorem 2
  • ...and 5 more