Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A block-coordinate descent framework for non-convex composite optimization. Application to sparse precision matrix estimation

Guillaume Lauga

TL;DR

The paper develops a general block-coordinate descent framework for non-convex composite optimization, targeting problems of the form \(\Psi(\mathbf{x}) = f(\mathbf{x}) + g(\mathbf{x})\) with \(g(\mathbf{x}) = \sum_{\ell=1}^L (\phi_\ell \circ \psi_\ell)(x_\ell)\). By constructing quadratic majorants for \(f\) under block-wise metrics and a tangent-based majorant for \(g\), the authors formulate versatile subproblems solvable via variable-metric forward-backward, proximal-Newton, or Gauss-Seidel updates, with convergence to critical points under the Kurdyka-\Lojasiewicz property. The framework is validated on non-convex sparse precision matrix estimation (non-convex Graphical Lasso), showing convergence guarantees and up to a 100-fold reduction in iterations to reach high-quality estimates. Across three solver families—Graphical ISTA, QUIC, and Primal-GLasso—the approach unifies and accelerates existing convex solvers in the non-convex setting, delivering substantial computational gains while maintaining estimation accuracy.

Abstract

Block-coordinate descent (BCD) is the method of choice to solve numerous large scale optimization problems, however their theoretical study for non-convex optimization, has received less attention. In this paper, we present a new block-coordinate descent (BCD) framework to tackle non-convex composite optimization problems, ensuring decrease of the objective function and convergence to a solution. This framework is general enough to include variable metric proximal gradient updates, proximal Newton updates, and alternated minimization updates. This generality allows to encompass three versions of the most used solvers in the sparse precision matrix estimation problem, deemed Graphical Lasso: graphical ISTA, Primal GLasso, and QUIC. We demonstrate the value of this new framework on non-convex sparse precision matrix estimation problems, providing convergence guarantees and up to a $100$-fold reduction in the number of iterations required to reach state-of-the-art estimation quality.

A block-coordinate descent framework for non-convex composite optimization. Application to sparse precision matrix estimation

TL;DR

The paper develops a general block-coordinate descent framework for non-convex composite optimization, targeting problems of the form \(\Psi(\mathbf{x}) = f(\mathbf{x}) + g(\mathbf{x})\) with \(g(\mathbf{x}) = \sum_{\ell=1}^L (\phi_\ell \circ \psi_\ell)(x_\ell)\). By constructing quadratic majorants for under block-wise metrics and a tangent-based majorant for , the authors formulate versatile subproblems solvable via variable-metric forward-backward, proximal-Newton, or Gauss-Seidel updates, with convergence to critical points under the Kurdyka-\Lojasiewicz property. The framework is validated on non-convex sparse precision matrix estimation (non-convex Graphical Lasso), showing convergence guarantees and up to a 100-fold reduction in iterations to reach high-quality estimates. Across three solver families—Graphical ISTA, QUIC, and Primal-GLasso—the approach unifies and accelerates existing convex solvers in the non-convex setting, delivering substantial computational gains while maintaining estimation accuracy.

Abstract

Block-coordinate descent (BCD) is the method of choice to solve numerous large scale optimization problems, however their theoretical study for non-convex optimization, has received less attention. In this paper, we present a new block-coordinate descent (BCD) framework to tackle non-convex composite optimization problems, ensuring decrease of the objective function and convergence to a solution. This framework is general enough to include variable metric proximal gradient updates, proximal Newton updates, and alternated minimization updates. This generality allows to encompass three versions of the most used solvers in the sparse precision matrix estimation problem, deemed Graphical Lasso: graphical ISTA, Primal GLasso, and QUIC. We demonstrate the value of this new framework on non-convex sparse precision matrix estimation problems, providing convergence guarantees and up to a -fold reduction in the number of iterations required to reach state-of-the-art estimation quality.
Paper Structure (40 sections, 15 theorems, 74 equations, 6 figures, 3 algorithms)

This paper contains 40 sections, 15 theorems, 74 equations, 6 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Composite optimization.
  3. Block-coordinate descent.
  4. Sparse precision matrix estimation.
  5. Contributions.
  6. Organization of the paper.
  7. Notations.
  8. Optimization background
  9. Subgradients
  10. The Kurdyka-Łojasiewicz (KŁ) property.
  11. Variable metric optimization
  12. Our block-coordinate descent framework for composite non-convex optimization.
  13. Assumptions
  14. Majorization of $f$ through variable metric.
  15. Majorization of the function $g$
  16. ...and 25 more sections

Key Result

Proposition 1

Chain rule VarAnalRockafellar. Let $f:\mathcal{H} \rightarrow \mathbb{R}$ be a differentiable function and $g:\mathcal{H} \rightarrow \mathbb{R}$, then we have $\partial(f+g) = \nabla f + \partial g$.

Figures (6)

  • Figure 1: Graphical-ISTA reconstruction for a sparsity level of $90\%$. Top left: maximum F$1$-score w.r.t. number of iterations between reweightings (log-scale). Bottom left: maximum F$1$-score w.r.t. total number of iterations. Top right: minimum NMSE w.r.t. number of iterations between reweightings (log-scale). Bottom right: minimum NMSE w.r.t. total number of iterations.
  • Figure 2: QUIC reconstruction for a sparsity level of $90\%$. Top left: maximum F$1$-score w.r.t. number of iterations between reweightings (log-scale). Bottom left: maximum F$1$-score w.r.t. total number of iterations. Top right: minimum NMSE w.r.t. number of iterations between reweightings (log-scale). Bottom right: minimum NMSE w.r.t. total number of iterations.
  • Figure 3: P-GLasso reconstruction for a sparsity level of $90\%$. Top left: maximum F$1$-score w.r.t. number of iterations between reweightings (log-scale). Bottom left: maximum F$1$-score w.r.t. total number of iterations. Top right: minimum NMSE w.r.t. number of iterations between reweightings (log-scale). Bottom right: minimum NMSE w.r.t. total number of iterations.
  • Figure 4: Graphical-ISTA reconstruction for a sparsity level of $75\%$. Top left: maximum F$1$-score w.r.t. number of iterations between reweightings (log-scale). Bottom left: maximum F$1$-score w.r.t. total number of iterations. Top right: minimum NMSE w.r.t. number of iterations between reweightings (log-scale). Bottom right: minimum NMSE w.r.t. total number of iterations.
  • Figure 5: QUIC reconstruction for a sparsity level of $75\%$. Top left: maximum F$1$-score w.r.t. number of iterations between reweightings (log-scale). Bottom left: maximum F$1$-score w.r.t. total number of iterations. Top right: minimum NMSE w.r.t. number of iterations between reweightings (log-scale). Bottom right: minimum NMSE w.r.t. total number of iterations.
  • ...and 1 more figures

Theorems & Definitions (33)

  • Definition 1
  • Proposition 1
  • Proposition 2
  • Proposition 3
  • Definition 2
  • Definition 3
  • Lemma 1
  • Remark 1
  • Definition 4
  • Proposition 4
  • ...and 23 more