Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Coordinate descent methods beyond smoothness and separability

Flavia Chorobura, Ion Necoara

TL;DR

The paper presents a unified smoothing framework for convex nonsmooth optimization that encompasses Moreau, Forward-Backward, Douglas-Rachford envelopes, and Nesterov smoothing. It analyzes (accelerated) randomized coordinate-descent methods on these smooth approximations, deriving sublinear rates in function value for the original objective and, under a $q$-growth condition, linear rates including restart variants. Additionally, it introduces a relative-smoothness along coordinates framework with a RRCD algorithm that achieves sublinear or linear rates depending on convexity and growth, and provides concrete implementation details for common relative kernels. The authors validate their theory with simulations on quadratic problems featuring $\ell_2$ and TV regularization, demonstrating the practical efficiency of accelerated and restart schemes. Overall, the work broadens the applicability of coordinate-descent methods to nonsmooth and nonseparable objectives through a versatile smoothing and relative-smoothness toolkit with strong convergence guarantees and empirical support.

Abstract

This paper deals with convex nonsmooth optimization problems. We introduce a general smooth approximation framework for the original function and apply random (accelerated) coordinate descent methods for minimizing the corresponding smooth approximations. Our framework covers the most important classes of smoothing techniques from the literature. Based on this general framework for the smooth approximation and using coordinate descent type methods we derive convergence rates in function values for the original objective. Moreover, if the original function satisfies a growth condition, then we prove that the smooth approximations also inherits this condition and consequently the convergence rates are improved in this case. We also present a relative randomized coordinate descent algorithm for solving nonseparable minimization problems with the objective function relative smooth along coordinates w.r.t. a (possibly nonseparable) differentiable function. For this algorithm we also derive convergence rates in the convex case and under the growth condition for the objective.

Coordinate descent methods beyond smoothness and separability

TL;DR

The paper presents a unified smoothing framework for convex nonsmooth optimization that encompasses Moreau, Forward-Backward, Douglas-Rachford envelopes, and Nesterov smoothing. It analyzes (accelerated) randomized coordinate-descent methods on these smooth approximations, deriving sublinear rates in function value for the original objective and, under a -growth condition, linear rates including restart variants. Additionally, it introduces a relative-smoothness along coordinates framework with a RRCD algorithm that achieves sublinear or linear rates depending on convexity and growth, and provides concrete implementation details for common relative kernels. The authors validate their theory with simulations on quadratic problems featuring and TV regularization, demonstrating the practical efficiency of accelerated and restart schemes. Overall, the work broadens the applicability of coordinate-descent methods to nonsmooth and nonseparable objectives through a versatile smoothing and relative-smoothness toolkit with strong convergence guarantees and empirical support.

Abstract

This paper deals with convex nonsmooth optimization problems. We introduce a general smooth approximation framework for the original function and apply random (accelerated) coordinate descent methods for minimizing the corresponding smooth approximations. Our framework covers the most important classes of smoothing techniques from the literature. Based on this general framework for the smooth approximation and using coordinate descent type methods we derive convergence rates in function values for the original objective. Moreover, if the original function satisfies a growth condition, then we prove that the smooth approximations also inherits this condition and consequently the convergence rates are improved in this case. We also present a relative randomized coordinate descent algorithm for solving nonseparable minimization problems with the objective function relative smooth along coordinates w.r.t. a (possibly nonseparable) differentiable function. For this algorithm we also derive convergence rates in the convex case and under the growth condition for the objective.
Paper Structure (23 sections, 17 theorems, 179 equations, 2 figures, 2 tables)

This paper contains 23 sections, 17 theorems, 179 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Smooth approximations for convex problems
  4. Moreau envelope
  5. Forward-Backward envelope
  6. Douglas-Rachford envelope
  7. Nesterov's smoothing
  8. Functional $q$-growth
  9. Moreau and Forward-Backward envelopes
  10. Nesterov's smoothing
  11. (Accelerated) coordinate descent methods
  12. Coordinate descent: convex case
  13. Accelerated coordinate descent: convex case
  14. Coordinate descent: $q$-growth case
  15. Restart accelerated coordinate descent: $q$-growth case
  16. ...and 8 more sections

Key Result

Lemma 1

Consider $f:\mathbb{R}^{n} \to \mathbb{R}$ a function and $X \subseteq \mathbb{R}^{n}$ a closed convex set. Assume that $f$ is convex along coordinates and the following inequality holds for all $x, x+U_{i} h\in X$ and $i=1,\cdots,N$ Then, we have: and

Figures (2)

  • Figure 1: Evolution of Algorithms 1 and 2 for four different smoothing techniques in function values along time for quadratic objective with $\ell_2$ regularization and $n=10^4$.
  • Figure 2: Evolution of Algorithms 1, 3 and their full variants for three different smoothing techniques in function values along time for quadratic objective with TV regularization and $n=10^4$.

Theorems & Definitions (37)

  • Definition 1
  • Definition 2
  • Lemma 1
  • proof
  • Theorem 1
  • proof
  • Theorem 2
  • proof
  • Corollary 1
  • proof
  • ...and 27 more