Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Coderivative-Based Newton Methods in Structured Nonconvex and Nonsmooth Optimization

Pham Duy Khanh, Boris S. Mordukhovich, Vo Thanh Phat

TL;DR

This work develops coderivative-based Newton methods for structured nonconvex and nonsmooth optimization, using second-order subdifferentials in place of Hessians and applying proximal mappings for non-smooth components. It introduces a globalization strategy via a line-search based on forward-backward envelopes, plus a globalized Newton variant, achieving local superlinear and global convergence under mild variational-analytic assumptions. The methods handle objective functions that are sums of a smooth part and a prox-regular non-smooth part (e.g., L0 penalties) and are demonstrated on nonconvex least squares with L0 penalties, Student's t-regression with L0 penalties, and image restoration, showing favorable performance against state-of-the-art second-order methods. Overall, the paper advances a variational-analysis–driven framework for efficient optimization of nonsmooth nonconvex problems with practical relevance to statistics and imaging.

Abstract

This paper proposes and develops new Newton-type methods to solve structured nonconvex and nonsmooth optimization problems with justifying their fast local and global convergence by means of advanced tools of variational analysis and generalized differentiation. The objective functions belong to a broad class of prox-regular functions with specification to constrained optimization of nonconvex structured sums. We also develop a novel line search method, which is an extension of the proximal gradient algorithm while allowing us to globalize the proposed coderivative-based Newton methods by incorporating the machinery of forward-backward envelopes. Applications and numerical experiments, which are provided for nonconvex least squares regression models, Student's t-regression, and image restoration problems, demonstrate the efficiency of the proposed algorithms

Coderivative-Based Newton Methods in Structured Nonconvex and Nonsmooth Optimization

TL;DR

This work develops coderivative-based Newton methods for structured nonconvex and nonsmooth optimization, using second-order subdifferentials in place of Hessians and applying proximal mappings for non-smooth components. It introduces a globalization strategy via a line-search based on forward-backward envelopes, plus a globalized Newton variant, achieving local superlinear and global convergence under mild variational-analytic assumptions. The methods handle objective functions that are sums of a smooth part and a prox-regular non-smooth part (e.g., L0 penalties) and are demonstrated on nonconvex least squares with L0 penalties, Student's t-regression with L0 penalties, and image restoration, showing favorable performance against state-of-the-art second-order methods. Overall, the paper advances a variational-analysis–driven framework for efficient optimization of nonsmooth nonconvex problems with practical relevance to statistics and imaging.

Abstract

This paper proposes and develops new Newton-type methods to solve structured nonconvex and nonsmooth optimization problems with justifying their fast local and global convergence by means of advanced tools of variational analysis and generalized differentiation. The objective functions belong to a broad class of prox-regular functions with specification to constrained optimization of nonconvex structured sums. We also develop a novel line search method, which is an extension of the proximal gradient algorithm while allowing us to globalize the proposed coderivative-based Newton methods by incorporating the machinery of forward-backward envelopes. Applications and numerical experiments, which are provided for nonconvex least squares regression models, Student's t-regression, and image restoration problems, demonstrate the efficiency of the proposed algorithms
Paper Structure (11 sections, 34 theorems, 242 equations, 3 figures, 2 tables, 4 algorithms)

This paper contains 11 sections, 34 theorems, 242 equations, 3 figures, 2 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries from Variational Analysis
  3. Coderivative-Based Newton Method for Subgradient Systems
  4. Local Convergence Analysis of Coderivative-Based Newton Method for Structured Nonconvex Optimization
  5. Generalized Line Search Proximal Gradient Method
  6. Globalization of Coderivative-Based Newton Method in Structured Nonconvex Optimization
  7. Applications and Numerical Experiments
  8. Nonconvex Least Square Regression Models
  9. Student’s t‑Regression with $\ell^0$-Penalty
  10. Restoration of Noisy Blurred Images
  11. Conclusions and Future Research

Key Result

Proposition 2.3

Let $\varphi:{\rm I\!R}^n\to\overline{{\rm I\!R}}$ be l.s.c. with $\bar{x}\in\hbox{\rm dom}\,\varphi$ and $\bar{v}\in \widehat{\partial}\varphi(\bar{x})$. Then for any $s \in {\rm I\!R}$, the following are equivalent: (i) $\varphi$ is variationally $s$-convex at $\bar{x}$ for $\bar{v}$. (ii) There a where $U_\varepsilon:=\{x\in U\;|\;\varphi(x)<\varphi(\bar{x})+\varepsilon\}$. (iii) There are neig

Figures (3)

  • Figure 1: The original and blurred cameramen test images
  • Figure 2: Recovered images with the three solvers with $\mu_0 =10^{-4}$, $\mu_2 = 5\times 10^{-3}$
  • Figure 3: Residual and CPU times of three solvers with $\mu_0 =10^{-4}$, $\mu_2 = 5\times 10^{-3}$

Theorems & Definitions (53)

  • Definition 2.1: metric regularity and subregularity of mappings
  • Definition 2.2: variationally convex functions
  • Proposition 2.3: subgradient characterizations of variational convexity
  • Proposition 2.4: Moreau envelopes and proximal mappings for prox-bounded functions
  • Proposition 2.5: Moreau envelopes and proximal mappings for prox-regular functions
  • Definition 2.6: tilt-stable local minimizers
  • Proposition 2.7: strong variational convexity and tilt stability
  • Proposition 2.8: second-order subdifferentials and quadratic bundles
  • Proposition 3.1: solvability of generalized Newton systems
  • Lemma 3.2: subspace property
  • ...and 43 more