Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A New Inexact Gradient Descent Method with Applications to Nonsmooth Convex Optimization

Pham Duy Khanh, Boris S. Mordukhovich, Dat Ba Tran

TL;DR

This work develops an inexact gradient framework for smooth and nonsmooth convex optimization. It introduces the inexact gradient descent (IGD) method for $\mathcal{C}^{1,1}$ objectives, proving convergence of both gradient norms to zero and of iterates to stationary points under the KL property, with rates tied to the KL exponent. The IGD framework is then used to derive two gradient-based inexact methods for nonsmooth convex optimization: GIPPM via Moreau envelopes and GIALM for convex programs with linear equality constraints, both inheriting global convergence and KL-driven rates. Numerical experiments on image deblurring and random problems show GIALM often outperforms classical IALM, highlighting the practical efficiency of controlled inexactness in proximal- and augmented-Lagrangian-based methods.

Abstract

The paper proposes and develops a novel inexact gradient method (IGD) for minimizing C1-smooth functions with Lipschitzian gradients, i.e., for problems of C1,1 optimization. We show that the sequence of gradients generated by IGD converges to zero. The convergence of iterates to stationary points is guaranteed under the Kurdyka- Lojasiewicz (KL) property of the objective function with convergence rates depending on the KL exponent. The newly developed IGD is applied to designing two novel gradient-based methods of nonsmooth convex optimization such as the inexact proximal point methods (GIPPM) and the inexact augmented Lagrangian method (GIALM) for convex programs with linear equality constraints. These two methods inherit global convergence properties from IGD and are confirmed by numerical experiments to have practical advantages over some well-known algorithms of nonsmooth convex optimization.

A New Inexact Gradient Descent Method with Applications to Nonsmooth Convex Optimization

TL;DR

This work develops an inexact gradient framework for smooth and nonsmooth convex optimization. It introduces the inexact gradient descent (IGD) method for objectives, proving convergence of both gradient norms to zero and of iterates to stationary points under the KL property, with rates tied to the KL exponent. The IGD framework is then used to derive two gradient-based inexact methods for nonsmooth convex optimization: GIPPM via Moreau envelopes and GIALM for convex programs with linear equality constraints, both inheriting global convergence and KL-driven rates. Numerical experiments on image deblurring and random problems show GIALM often outperforms classical IALM, highlighting the practical efficiency of controlled inexactness in proximal- and augmented-Lagrangian-based methods.

Abstract

The paper proposes and develops a novel inexact gradient method (IGD) for minimizing C1-smooth functions with Lipschitzian gradients, i.e., for problems of C1,1 optimization. We show that the sequence of gradients generated by IGD converges to zero. The convergence of iterates to stationary points is guaranteed under the Kurdyka- Lojasiewicz (KL) property of the objective function with convergence rates depending on the KL exponent. The newly developed IGD is applied to designing two novel gradient-based methods of nonsmooth convex optimization such as the inexact proximal point methods (GIPPM) and the inexact augmented Lagrangian method (GIALM) for convex programs with linear equality constraints. These two methods inherit global convergence properties from IGD and are confirmed by numerical experiments to have practical advantages over some well-known algorithms of nonsmooth convex optimization.
Paper Structure (11 sections, 9 theorems, 73 equations, 3 figures, 1 table)

This paper contains 11 sections, 9 theorems, 73 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Preliminaries and Discussions
  3. Inexact Gradient Descent Method
  4. Algorithm Formulation and Motivating Examples
  5. Convergence Analysis
  6. Gradient-Based Inexact Methods in Nonsmooth Convex Optimization
  7. Gradient-Based Inexact Proximal Point Method
  8. Gradient-Based Inexact Augmented Lagrangian Method
  9. Numerical Experiments
  10. An Example in Image Processing
  11. Randomly Generated Examples

Key Result

Proposition 2.3

Let $f:{\rm I\!R}^n\rightarrow{\rm I\!R}$ be a $\mathcal{C}^1$-smooth function, and let the following conditions hold along a sequence of iterates $\left\{x^k\right\}\subset{\rm I\!R}^n$ for the function $f$: If $\bar{x}$ is an accumulation point of $\left\{x^k\right\}$ and $f$ satisfies the KL property at $\bar{x}$, then $x^k\rightarrow\bar{x}$ as $k\to\infty$.

Figures (3)

  • Figure 2: Deblurring of the cameraman
  • Figure 3: Function values (left) and total number of iterations in subproblems (right)
  • Figure 4: Value of residual $\eta_k$ (left) and error $\omega_k$ (right) in Test 12

Theorems & Definitions (18)

  • Definition 2.1
  • Remark 2.2
  • Proposition 2.3
  • Proposition 2.4
  • Remark 3.1
  • Example 1: Gradient approximation methods
  • Example 2: Moreau envelopes
  • Theorem 3.2
  • Remark 3.3
  • Theorem 3.4
  • ...and 8 more