Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ItsOPT: An inexact two-level smoothing framework for nonconvex optimization via high-order Moreau envelope

Alireza Kabgani, Masoud Ahookhosh

TL;DR

This paper addresses nonsmooth, nonconvex optimization by introducing ItsOPT, an inexact two-level smoothing framework that leverages a high-order Moreau envelope (HOME) to create a smooth surrogate and an inexact proximal oracle derived from a lower-level high-order proximal problem (HOPE). The framework enables upper-level first-/second-order methods to operate with an inexact oracle, and includes a Boosted HiPPA (nonmonotone line-search) to efficiently reach proximal fixed points. The authors establish global convergence under the KL property and prove a novel linear convergence result: for any KL exponent $\theta\in(0,1)$, choosing the order as $p=\frac{1}{1-\theta}$ yields linear convergence, which is a first for KL functions in this setting. Preliminary numerical experiments on robust low-rank matrix recovery demonstrate that Boosted HiPPA outperforms existing approaches and validate the theoretical findings, indicating strong practical potential for broad nonsmooth, nonconvex problems.

Abstract

This paper introduces ItsOPT, an inexact two-level smoothing optimization framework designed to find first-order critical points of nonsmooth and nonconvex functions. The framework involves two levels of methodologies: at the upper level, a zero-, first-, or second-order method will be tailored to minimize a smooth approximation; at the lower level, the high-order proximal auxiliary problems will be solved inexactly, generating an inexact oracle for the smooth function. As a smoothing technique, we here introduce the high-order Moreau envelope (HOME) and study its fundamental features under standard assumptions. Next, introducing a boosted high-order proximal-point algorithm (Boosted HiPPA) at the upper level using the inexact oracle from the lower level leads to an instance of ItsOPT. Global convergence rates are established under the Kurdyka-Łojasiewicz (KL) property of the cost and envelope functions, along with some reasonable conditions for the accuracy of the proximal terms. surprisingly, for any KL exponent $θ\in (0,1)$ of the original cost, setting the regularization order $p=\frac{1}{1-θ}$ ensures that Boosted HiPPA converges linearly to a proximal fixed point, which is the first algorithm with this property for KL functions. Preliminary numerical experiments on a robust low-rank matrix recovery problem indicate a promising performance of the proposed algorithm, validating our theoretical foundations.

ItsOPT: An inexact two-level smoothing framework for nonconvex optimization via high-order Moreau envelope

TL;DR

This paper addresses nonsmooth, nonconvex optimization by introducing ItsOPT, an inexact two-level smoothing framework that leverages a high-order Moreau envelope (HOME) to create a smooth surrogate and an inexact proximal oracle derived from a lower-level high-order proximal problem (HOPE). The framework enables upper-level first-/second-order methods to operate with an inexact oracle, and includes a Boosted HiPPA (nonmonotone line-search) to efficiently reach proximal fixed points. The authors establish global convergence under the KL property and prove a novel linear convergence result: for any KL exponent , choosing the order as yields linear convergence, which is a first for KL functions in this setting. Preliminary numerical experiments on robust low-rank matrix recovery demonstrate that Boosted HiPPA outperforms existing approaches and validate the theoretical findings, indicating strong practical potential for broad nonsmooth, nonconvex problems.

Abstract

This paper introduces ItsOPT, an inexact two-level smoothing optimization framework designed to find first-order critical points of nonsmooth and nonconvex functions. The framework involves two levels of methodologies: at the upper level, a zero-, first-, or second-order method will be tailored to minimize a smooth approximation; at the lower level, the high-order proximal auxiliary problems will be solved inexactly, generating an inexact oracle for the smooth function. As a smoothing technique, we here introduce the high-order Moreau envelope (HOME) and study its fundamental features under standard assumptions. Next, introducing a boosted high-order proximal-point algorithm (Boosted HiPPA) at the upper level using the inexact oracle from the lower level leads to an instance of ItsOPT. Global convergence rates are established under the Kurdyka-Łojasiewicz (KL) property of the cost and envelope functions, along with some reasonable conditions for the accuracy of the proximal terms. surprisingly, for any KL exponent of the original cost, setting the regularization order ensures that Boosted HiPPA converges linearly to a proximal fixed point, which is the first algorithm with this property for KL functions. Preliminary numerical experiments on a robust low-rank matrix recovery problem indicate a promising performance of the proposed algorithm, validating our theoretical foundations.

Paper Structure

This paper contains 13 sections, 13 theorems, 78 equations, 1 figure, 2 algorithms.

Table of Contents

  1. Introduction
  2. Contribution and related works
  3. Paper organization
  4. Preliminaries and notations
  5. ItsOPT: inexact two-level smoothing framework
  6. High-order Moreau envelope in the nonconvex setting
  7. Inexact oracle for HOME
  8. Boosted high-order inexact proximal-point algorithm
  9. Global and linear convergence
  10. Preliminary numerical experiments
  11. Implementation issues
  12. Application to robust low-rank matrix recovery
  13. Conclusions

Key Result

Lemma 2.1

Assume that $a, b\in \mathbb{R}^n$.

Figures (1)

  • Figure 1: The sequences of residual $\left\{\Vert R_\gamma^{\varepsilon_k}(x^k)\Vert\right\}_{k\geq 0}$ and function values $\left\{\Phi(x^k)\right\}_{k\geq 0}$ and $\left\{\Psi(x^k)\right\}_{k\geq 0}$ versus iterations for the problems \ref{['eq:Experiments:optmodel']} and \ref{['eq:Experiments:optmodel2']}.

Theorems & Definitions (35)

  • Lemma 2.1: Basic inequalities I
  • Proof 1
  • Definition 2.2: Kurdyka-Łojasiewicz property
  • Remark 2.3
  • Lemma 2.4: Uniformized KL property
  • Proof 2
  • Definition 3.1: High-order proximal operator and Moreau envelope
  • Definition 3.3: High-order prox-boundedness
  • Theorem 3.4: Basic properties of HOME and HOPE
  • Proof 3
  • ...and 25 more