Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Box-constrained L0 Bregman-relaxations

Mhamed Essafri, Luca Calatroni, Emmanuel Soubies

TL;DR

This work tackles the NP-hard problem of L0-regularized optimization under box constraints by introducing a box-constrained L0 Bregman relaxation (B-rex) that yields an exact continuous surrogate $J^{l,u}_\Psi$ retaining all global minimizers of the original problem $J_0$ while reducing spurious local minima.The authors derive a closed-form, separable expression for the relaxation $B_\Psi^{l,u}$, establish an exact-relaxation condition, and develop practical solvers including Forward-Backward Splitting (FBS) and Iteratively Reweighted L1 (IRL1) that leverage the proximal structure of the relaxation.Experiments on synthetic data with least-squares and logistic/KL-style fidelities compare against Branch-and-Bound and Iterative Hard Thresholding, showing the relaxed formulations yield near-global-optimal solutions with substantially lower computation times, and that the box-constrained approach remains effective as sparsity varies.Overall, the box-constrained B-rex framework provides a scalable, theoretically grounded approach to sparse recovery with non-quadratic data terms, preserving global optima and enabling efficient optimization for practical signal processing and machine learning tasks.

Abstract

Regularization using the L0 pseudo-norm is a common approach to promote sparsity, with widespread applications in machine learning and signal processing. However, solving such problems is known to be NP-hard. Recently, the L0 Bregman relaxation (B-rex) has been introduced as a continuous, non-convex approximation of the L0 pseudo-norm. Replacing the L0 term with B-rex leads to exact continuous relaxations that preserve the global optimum while simplifying the optimization landscape, making non-convex problems more tractable for algorithmic approaches. In this paper, we focus on box-constrained exact continuous Bregman relaxations of L0-regularized criteria with general data terms, including least-squares, logistic regression, and Kullback-Leibler fidelities. Experimental results on synthetic data, compared with Branch-and-Bound methods, demonstrate the effectiveness of the proposed relaxations.

Box-constrained L0 Bregman-relaxations

TL;DR

This work tackles the NP-hard problem of L0-regularized optimization under box constraints by introducing a box-constrained L0 Bregman relaxation (B-rex) that yields an exact continuous surrogate $J^{l,u}_\Psi$ retaining all global minimizers of the original problem $J_0$ while reducing spurious local minima.The authors derive a closed-form, separable expression for the relaxation $B_\Psi^{l,u}$, establish an exact-relaxation condition, and develop practical solvers including Forward-Backward Splitting (FBS) and Iteratively Reweighted L1 (IRL1) that leverage the proximal structure of the relaxation.Experiments on synthetic data with least-squares and logistic/KL-style fidelities compare against Branch-and-Bound and Iterative Hard Thresholding, showing the relaxed formulations yield near-global-optimal solutions with substantially lower computation times, and that the box-constrained approach remains effective as sparsity varies.Overall, the box-constrained B-rex framework provides a scalable, theoretically grounded approach to sparse recovery with non-quadratic data terms, preserving global optima and enabling efficient optimization for practical signal processing and machine learning tasks.

Abstract

Regularization using the L0 pseudo-norm is a common approach to promote sparsity, with widespread applications in machine learning and signal processing. However, solving such problems is known to be NP-hard. Recently, the L0 Bregman relaxation (B-rex) has been introduced as a continuous, non-convex approximation of the L0 pseudo-norm. Replacing the L0 term with B-rex leads to exact continuous relaxations that preserve the global optimum while simplifying the optimization landscape, making non-convex problems more tractable for algorithmic approaches. In this paper, we focus on box-constrained exact continuous Bregman relaxations of L0-regularized criteria with general data terms, including least-squares, logistic regression, and Kullback-Leibler fidelities. Experimental results on synthetic data, compared with Branch-and-Bound methods, demonstrate the effectiveness of the proposed relaxations.

Paper Structure

This paper contains 17 sections, 3 theorems, 21 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Related Works
  3. Contributions and Outline
  4. Notations
  5. Box-Constrained B-rex
  6. Exact Relaxations Properties
  7. Minimizing $J_\Psi^{l,u}$
  8. Forward-Backward Splitting Algorithm
  9. Iteratively Reweighted $\ell_1$
  10. Experiments
  11. Data Generation
  12. Algorithmic Setting and Parameter Selection
  13. Results and Discussion
  14. Conclusion
  15. Proof of Proposition \ref{['prop:main_prop']}
  16. ...and 2 more sections

Key Result

Proposition 1

For all $n \in [N]$, let $\alpha^-_n \leq 0$ and $\alpha^+_n \geq 0$ be such that $[\alpha_n^-, \alpha_n^+]$ defines the set of sublevels $\lambda_0$ of $d_{\psi_n}(0, \cdot)$. Then, for every $\mathbf{x} \in [l,u]^N$, we have $B_\Psi^{l,u}(\mathbf{x}) = \sum_{n=1}^N \beta^{l,u}_{\psi_n}(x_n),$ wher where $\eta_n^- = \max\{\alpha_n^-, l\}, \quad \eta^+_n = \min\{\alpha_n^+, u\}$. Moreover, for $l\

Figures (4)

  • Figure 1: Illustration of box-constrained B-rex when (left) $[l,u] \supseteq [\alpha^-,\alpha^+]$ and (right) $[l,u] \subseteq [\alpha^-,\alpha^+]$.
  • Figure 2: LS: (ordered) values $J_0(\hat{\mathbf{x}})$ obtained by each method along the $20$ problem instances. For the left plot $\lambda_0 = 2 \times 10^{-2} F_\mathbf{y}(\textbf{0})$ and $k^*=10$. For the right plot $\lambda_0 = 5 \times 10^{-3}F_\mathbf{y}(\textbf{0})$ and $k^*=25$. Finally, $[l,u] = [-1.5, 1.5]$, $\rho=0.9$ and $\mathrm{SNR} = 10$.
  • Figure 3: LR: (ordered) values $J_0(\hat{\mathbf{x}})$ obtained by each method along the $20$ problem instances. For the left plot $\lambda_0 = 2.5 \times 10^{-2} F_\mathbf{y}(\textbf{0})$ and $k^*=7$. For the right $\lambda_0 = 1.5 \times 10^{-2} F_\mathbf{y}(\textbf{0})$ and $k^*=25$. Finally $[l,u] = [-1, 1]$, $\lambda_2 = 1$, $\rho = 0.9$, $s=1$.
  • Figure 4: LS: (ordered) values of $J_0(\hat{\mathbf{x}})$ obtained by each method along the $20$ problem instances, with $\lambda_0 = 5.5 \times 10^{-3} F_\mathbf{y}(\textbf{0})$, $\lambda_2 = 2$, $[l,u] = [-1.5, 1.5]$, $k^* = 25$, $\rho = 0.9$ and $\mathrm{SNR} = 10$.

Theorems & Definitions (7)

  • Remark 1
  • Proposition 1: Closed form expression of $B_\Psi^{l,u}$
  • proof
  • Theorem 2: Exact relaxation property
  • proof
  • Proposition 3
  • proof