Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Random coordinate descent methods for nonseparable composite optimization

Flavia Chorobura, Ion Necoara

TL;DR

This work develops two coordinate descent frameworks, CPG and CGD, for large-scale composite optimization F(x)=f(x)+\psi(x) where f has block Lipschitz gradients and \psi is differentiable but potentially nonseparable. It provides descent-guaranteeing adaptive stepsizes, complete convergence analyses in convex and nonconvex settings, and KL-based rates, for both random and cyclic coordinate selection. The authors also establish sufficient conditions for bounded iterates and demonstrate the methods on cubic-Newton subproblems and penalized matrix factorization, showing competitive or superior performance to existing approaches. The results advance scalable optimization by handling nonseparability in the smooth term with provable guarantees and practical efficiency.

Abstract

In this paper we consider large-scale composite optimization problems having the objective function formed as a sum of two terms (possibly nonconvex), one has (block) coordinate-wise Lipschitz continuous gradient and the other is differentiable but nonseparable. Under these general settings we derive and analyze two new coordinate descent methods. The first algorithm, referred to as coordinate proximal gradient method, considers the composite form of the objective function, while the other algorithm disregards the composite form of the objective and uses the partial gradient of the full objective, yielding a coordinate gradient descent scheme with novel adaptive stepsize rules. We prove that these new stepsize rules make the coordinate gradient scheme a descent method, provided that additional assumptions hold for the second term in the objective function. We present a complete worst-case complexity analysis for these two new methods in both, convex and nonconvex settings, provided that the (block) coordinates are chosen random or cyclic. Preliminary numerical results also confirm the efficiency of our two algorithms on practical problems.

Random coordinate descent methods for nonseparable composite optimization

TL;DR

This work develops two coordinate descent frameworks, CPG and CGD, for large-scale composite optimization F(x)=f(x)+\psi(x) where f has block Lipschitz gradients and \psi is differentiable but potentially nonseparable. It provides descent-guaranteeing adaptive stepsizes, complete convergence analyses in convex and nonconvex settings, and KL-based rates, for both random and cyclic coordinate selection. The authors also establish sufficient conditions for bounded iterates and demonstrate the methods on cubic-Newton subproblems and penalized matrix factorization, showing competitive or superior performance to existing approaches. The results advance scalable optimization by handling nonseparability in the smooth term with provable guarantees and practical efficiency.

Abstract

In this paper we consider large-scale composite optimization problems having the objective function formed as a sum of two terms (possibly nonconvex), one has (block) coordinate-wise Lipschitz continuous gradient and the other is differentiable but nonseparable. Under these general settings we derive and analyze two new coordinate descent methods. The first algorithm, referred to as coordinate proximal gradient method, considers the composite form of the objective function, while the other algorithm disregards the composite form of the objective and uses the partial gradient of the full objective, yielding a coordinate gradient descent scheme with novel adaptive stepsize rules. We prove that these new stepsize rules make the coordinate gradient scheme a descent method, provided that additional assumptions hold for the second term in the objective function. We present a complete worst-case complexity analysis for these two new methods in both, convex and nonconvex settings, provided that the (block) coordinates are chosen random or cyclic. Preliminary numerical results also confirm the efficiency of our two algorithms on practical problems.
Paper Structure (15 sections, 16 theorems, 155 equations, 3 figures, 1 table)

This paper contains 15 sections, 16 theorems, 155 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Assumptions/setup
  4. KL property
  5. A coordinate proximal gradient algorithm
  6. A coordinate gradient descent algorithm
  7. Sufficient conditions for bounded iterates
  8. Convergence analysis: nonconvex case
  9. Sublinear convergence
  10. Better convergence under KL
  11. Convergence analysis: convex case
  12. Numerical simulations
  13. Cubic Newton
  14. Matrix factorization
  15. Conclusions

Key Result

Lemma 2.3

\newlabellemma:MVI0 Let $G: \mathbb{R}^{n} \to \mathbb{R}^{m}$ be a continuously differentiable function and $J: \mathbb{R}^{n} \to \mathbb{R}^{m\times n}$ be its Jacobian. Consider $U\in\mathbb{R}^{n\times r}$ a fixed matrix and $x, x+Ud \in \mathbb{R}^{n}$, with $d \in \mathbb{R}^{r}$. Then, t

Figures (3)

  • Figure 1: Behaviour along time (sec) of RCPG, RCGD-2, GD-2, algorithm Nes:19 and power method on group SchenkIBMNA, matrix c-30: left - $\|Ax_k- \lambda_k x_k\|$, right - $\lambda_k$.
  • Figure 2: CCGD and BMM on SalinasA: left - function values, right - orthogonality error
  • Figure 3: CCGD and BMM on Indian Pines: left - function values, right - orthogonality error.

Theorems & Definitions (36)

  • Definition 2.2
  • Lemma 2.3
  • Definition 2.4
  • Lemma 3.1
  • Proof 1
  • Lemma 3.2
  • Proof 2
  • Remark 3.3
  • Lemma 4.2
  • Proof 3
  • ...and 26 more