Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Projected proximal gradient trust-region algorithm for nonsmooth optimization

Minh N. Dao, Hung M. Phan, Lindon Roberts

TL;DR

This paper tackles nonsmooth composite optimization with $F(x)=f(x)+h(x)$ by extending trust-region methods to settings with potentially unbounded Hessian growth, and proves corresponding worst-case complexity bounds. It introduces a simple yet effective projected proximal gradient (PPG) subproblem solver that enforces the trust-region constraint at the end and achieves guaranteed decrease, matching the performance of more complex solvers in practice. Numerical experiments on 154 CUTEst problems show PPG often outperforms SPG at high accuracy, while SPG may be faster at very low accuracy; increasing the subproblem budget further enhances PPG's advantage. Overall, the work advances the theory and practice of nonsmooth, nonconvex optimization by unifying unbounded-Hessian analysis with a lightweight, robust subproblem solver that scales well in accuracy and problem size.

Abstract

We consider trust-region methods for solving optimization problems where the objective is the sum of a smooth, nonconvex function and a nonsmooth, convex regularizer. We extend the global convergence theory of such methods to include worst-case complexity bounds in the case of unbounded model Hessian growth, and introduce a new, simple nonsmooth trust-region subproblem solver based on combining several iterations of proximal gradient descent with a single projection into the trust region, which meets the sufficient descent requirements for algorithm convergence and has promising numerical results.

Projected proximal gradient trust-region algorithm for nonsmooth optimization

TL;DR

This paper tackles nonsmooth composite optimization with by extending trust-region methods to settings with potentially unbounded Hessian growth, and proves corresponding worst-case complexity bounds. It introduces a simple yet effective projected proximal gradient (PPG) subproblem solver that enforces the trust-region constraint at the end and achieves guaranteed decrease, matching the performance of more complex solvers in practice. Numerical experiments on 154 CUTEst problems show PPG often outperforms SPG at high accuracy, while SPG may be faster at very low accuracy; increasing the subproblem budget further enhances PPG's advantage. Overall, the work advances the theory and practice of nonsmooth, nonconvex optimization by unifying unbounded-Hessian analysis with a lightweight, robust subproblem solver that scales well in accuracy and problem size.

Abstract

We consider trust-region methods for solving optimization problems where the objective is the sum of a smooth, nonconvex function and a nonsmooth, convex regularizer. We extend the global convergence theory of such methods to include worst-case complexity bounds in the case of unbounded model Hessian growth, and introduce a new, simple nonsmooth trust-region subproblem solver based on combining several iterations of proximal gradient descent with a single projection into the trust region, which meets the sufficient descent requirements for algorithm convergence and has promising numerical results.
Paper Structure (12 sections, 19 theorems, 75 equations, 1 figure, 1 table, 2 algorithms)

This paper contains 12 sections, 19 theorems, 75 equations, 1 figure, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Existing Works
  3. Contributions
  4. Organization and Notation
  5. Preliminaries
  6. Trust-Region Algorithm and Worst-Case Complexity
  7. Main Algorithm
  8. Worst-Case Complexity
  9. Projected Proximal Gradient Subproblem Solver
  10. Numerical Experiments
  11. Conclusion and Future Work
  12. List of Test Problems

Key Result

Proposition 2.1

Let $h\colon \mathbb R^d\to (-\infty, +\infty]$ be proper and $f\colon \mathbb R^d\to (-\infty, +\infty]$ be differentiable at $x\in\operatorname{dom} h$. Then

Figures (1)

  • Figure 1: Comparison of PPG (Algorithm \ref{['alg_ppg']}) and SPG Baraldi2023a subproblem solvers with increasing maximum iterations $N\in\{15,30,50\}$ in the trust-region method Algorithm \ref{['alg_nonsmooth_tr']}.

Theorems & Definitions (41)

  • Proposition 2.1
  • proof
  • Proposition 2.2
  • proof
  • Proposition 2.3
  • proof
  • Example 2.4
  • Proposition 2.5
  • proof
  • Theorem 3.4
  • ...and 31 more