Subgradient Regularization: A Descent-Oriented Subgradient Method for Nonsmooth Optimization

TL;DR

This work develops a unifying descent-oriented framework for nonsmooth optimization, centered on descent-oriented subdifferentials that stabilize directions and guarantee convergence to stationary points. It introduces subgradient regularization (SRDescent) to construct descent directions from a single reference point for broad marginal problems, and shows that for compositions $f=h\circ c$ the method recovers the prox-linear update with a dual interpretation. An adaptive variant (SRDescent-adapt) leverages two regularization streams to preserve effective stepsizes and achieve local linear convergence under suitable regularity, while maintaining subsequential convergence in general. Numerical experiments across finite max/min of quadratics and marginal problems with varying feasible sets demonstrate robust performance and, in many cases, linear convergence behavior, highlighting the practical impact of the proposed framework for challenging nonsmooth optimization tasks.

Abstract

In nonsmooth optimization, a negative subgradient is not necessarily a descent direction, making the design of convergent descent methods based on zeroth-order and first-order information a challenging task. The well-studied bundle methods and gradient sampling algorithms construct descent directions by aggregating subgradients at nearby points in seemingly different ways, and are often complicated or lack deterministic guarantees. In this work, we identify a unifying principle behind these approaches, and develop a general framework of descent methods under the abstract principle that provably converge to stationary points. Within this framework, we introduce a simple yet effective technique, called subgradient regularization, to generate stable descent directions for a broad class of nonsmooth marginal functions, including finite maxima or minima of smooth functions. When applied to the composition of a convex function with a smooth map, the method naturally recovers the prox-linear method and, as a byproduct, provides a new dual interpretation of this classical algorithm. Numerical experiments demonstrate the effectiveness of our methods on several challenging classes of nonsmooth optimization problems, including the minimization of Nesterov's nonsmooth Chebyshev-Rosenbrock function.

Figures (7)

• Figure 1: Contour plots and descent directions of Nesterov's nonsmooth Chebyshev-Rosenbrock function $f(x) = \frac{1}{4}(x_{1}-1)^{2} + |x_{2}-2(x_{1})^{2}+1|$, which can be expressed as the maximum of two smooth functions $f_{1}$ and $f_{2}$. In each plot, dashed arrows show the descent direction computed by the respective methods, while different markers indicate the points used to construct these directions. Left: In gradient sampling algorithm, four nearby sample points (crosses) around $x^{1}$ are used to compute a minimal norm convex combination of gradients. Middle: In bundle method, gradients from previous iterates $\nabla f_{1}(x^{1})$ and $\nabla f_{2}(x^{2})$ are aggregated to form a descent direction at $x^{2}$. Right: SRDescent directly combines gradients $\nabla f_{1}(x^{1})$ and $\nabla f_{2}(x^{1})$ and their function values at the current iterate $x^{1}$.
• Figure 2: Performance of SRDescent, gradient sampling burke2005robust, and a bundle method specially designed for difference-of-convex functions de2019proximal on Nesterov's nonsmooth Chebyshev-Rosenbrock function $f(x) = \frac{1}{4}(x_{1}- 1)^{2}+ \sum^{6}_{i=1} |x_{i+1}-2(x_{i})^2+1|$. The gradient sampling algorithm terminates due to the failure of line search.
• Figure 3: Performance on the max of convex quadratics functions with $n=200$ and $m \in \{10, 50, 100, 200\}$, all initialized from the same random points. For the Polyak method, we plot the best objective value after $k$ oracle calls, i.e., $\min_{1 \leq i \leq k} f(x^k)$ rather than $f(x^k)$. In cases $m \in \{10, 50, 100\}$, GS terminates early due to the line search failures.
• Figure 4: Run time and oracle calls on the finite min of convex quadratic functions over $100$ randomly generated instances with $n=d=300$ and varying numbers of pieces $m$. The lines represent the median and shaded areas indicate inter-quartiles. NTDescent terminates when the objective gap $f(x^k) - f^\ast$ is below $10^{-6}$ and other methods terminate when the objective gap is below $10^{-8}$.
• Figure 5: Performance on the finite min of convex quadratic functions for $n=d=300$ and varying numbers of pieces $m \in \{10, 50, 100\}$, all initialized from the same randomly generated points. For Polyak, we report the best objective value after $k$ oracle calls, i.e., $\min_{1 \leq i \leq k} f(x^k)$, rather than $f(x^k)$. For all cases, BFGS terminates early due to the line search failures.

Theorems & Definitions (50)

• Definition 1: Descent-oriented subdifferentials
• Remark 1: Scaling
• Remark 2
• Proposition 1: Characterization of stationary points
• proof
• Proposition 2: Existence of descent directions
• Remark 3
• proof
• Proposition 3
• proof