Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Binno: A 1st-order method for Bi-level Nonconvex Nonsmooth Optimization for Matrix Factorizations

Laura Selicato, Flavia Esposito, Andersen Ang

TL;DR

Binno addresses nonconvex, nonsmooth bi-level optimization in matrix factorization by extending PALM into a bi-level setting. It performs blockwise proximal-gradient updates for both the upper and lower levels to generate two tentative iterates, then forms a calibrated convex combination with weights $\alpha$ and $\beta$ and a line search to enforce descent on both levelwise objectives, with theoretical conditions guaranteeing the existence of such weights. The paper provides descent proofs, explicit bounds for $\alpha$ and $\beta$, and practical updates for sparse low-rank factorization using $\ell_1$ and nuclear norms, validated on synthetic data and real traffic video where Binno attains lower reconstruction error and higher PSNR. This work delivers a modular first-order framework with provable descent for challenging bi-level, nonconvex nonsmooth problems and demonstrates practical impact in structured matrix factorization tasks.

Abstract

In this work, we develop a method for nonconvex, nonsmooth bi-level optimization and we introduce Binno, a first order method that leverages proximal constructions together with carefully designed descent conditions and variational analysis. Within this framework, Binno provably enforces a descent property for the overall objective surrogate associated with the bi-level problem. Each iteration performs blockwise proximal-gradient updates for the upper and the lower problems separately and then forms a calibrated, block-diagonal convex combination of the two tentative iterates. A linesearch selects the combination weights to enforce simultaneous descent of both level-wise objectives, and we establish conditions guaranteeing the existence of such weights together with descent directions induced by the associated proximal-gradient maps. We also apply Binno in the context of sparse low-rank factorization, where the upper level uses elementwise $\ell_1$ penalties and the lower level uses nuclear norms, coupled via a Frobenius data term. We test Binno on synthetic matrix and a real traffic-video dataset, attaining lower relative reconstruction error and higher peak signal-to-noise ratio than some standard methods.

Binno: A 1st-order method for Bi-level Nonconvex Nonsmooth Optimization for Matrix Factorizations

TL;DR

Binno addresses nonconvex, nonsmooth bi-level optimization in matrix factorization by extending PALM into a bi-level setting. It performs blockwise proximal-gradient updates for both the upper and lower levels to generate two tentative iterates, then forms a calibrated convex combination with weights and and a line search to enforce descent on both levelwise objectives, with theoretical conditions guaranteeing the existence of such weights. The paper provides descent proofs, explicit bounds for and , and practical updates for sparse low-rank factorization using and nuclear norms, validated on synthetic data and real traffic video where Binno attains lower reconstruction error and higher PSNR. This work delivers a modular first-order framework with provable descent for challenging bi-level, nonconvex nonsmooth problems and demonstrates practical impact in structured matrix factorization tasks.

Abstract

In this work, we develop a method for nonconvex, nonsmooth bi-level optimization and we introduce Binno, a first order method that leverages proximal constructions together with carefully designed descent conditions and variational analysis. Within this framework, Binno provably enforces a descent property for the overall objective surrogate associated with the bi-level problem. Each iteration performs blockwise proximal-gradient updates for the upper and the lower problems separately and then forms a calibrated, block-diagonal convex combination of the two tentative iterates. A linesearch selects the combination weights to enforce simultaneous descent of both level-wise objectives, and we establish conditions guaranteeing the existence of such weights together with descent directions induced by the associated proximal-gradient maps. We also apply Binno in the context of sparse low-rank factorization, where the upper level uses elementwise penalties and the lower level uses nuclear norms, coupled via a Frobenius data term. We test Binno on synthetic matrix and a real traffic-video dataset, attaining lower relative reconstruction error and higher peak signal-to-noise ratio than some standard methods.
Paper Structure (19 sections, 17 theorems, 73 equations, 5 figures, 2 tables, 2 algorithms)

This paper contains 19 sections, 17 theorems, 73 equations, 5 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Contribution and Paper Organization
  3. Background and mathematical tools
  4. Proximal gradient update
  5. Proximal Alternating Linearized Minimization (PALM)
  6. Sequential Averaging Method (SAM)
  7. The proposed method Binno
  8. Theory of Binno
  9. Application to Sparse Low Rank Factorization
  10. X-update
  11. Y-update
  12. Useful Theoretical Results
  13. Finding the constant $\alpha$
  14. Finding the constant $\beta$
  15. Numerical Experiments
  16. ...and 4 more sections

Key Result

Lemma 1

If $f$ is convex, proper and lower semicontinuous, then $M_{\nu f}\in C^1$ and for all $\bm{x}\in\mathbb{R}^{n}$, we have $\nabla M_{\nu f}=\frac{1}{\nu}(\bm{x}-\textrm{prox}_{\nu f}(\bm{x}))$. Consequently, its gradient is $1/\nu$ Lipschitz continuous.

Figures (5)

  • Figure 1: The structure of Binno. GD stands for gradient descent. In this work, our goal is to answer how to deal with question mark.
  • Figure 2: Synthetic experiment: (left) sparse factors and resulting matrix; (right) Objective bi-level functions convergence with respect to iterate.
  • Figure 3: Objective bi-level function convergence wrt iterates for the real dataset.
  • Figure 4: (a) Original clip (clip 5 as example) and its noisy observation; (b) Low-rank $\bm{L}$ and sparse $\bm{S}$ matrices into which the observed matrix is decomposed for all the tested algorithms.
  • Figure 5: Low-rank $\bm{L}$ and sparse $\bm{S}$ matrices into which the observed matrix is decomposed for all the tested algorithms, for 6 different clips.

Theorems & Definitions (35)

  • Lemma 1: Theorem 12.30, Bauschke2017
  • Remark
  • Theorem 1
  • Lemma 2
  • proof
  • Lemma 3
  • proof
  • Lemma 4
  • proof
  • Lemma 5
  • ...and 25 more