Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Efficient Alternating Algorithm for ReLU-based Symmetric Matrix Decomposition

Qingsong Wang

TL;DR

The paper tackles low-rank symmetric matrix decomposition for non-negative, sparse data by formulating $M \approx \max(0,X)$ with $X=UU^{T}$. It introduces ReLU-NSMD and an accelerated alternating partial Bregman (AAPB) method that alternates between a closed-form $W$-update and a $U$-update solved via a Bregman proximal gradient step under the $L$-smad condition, achieving convergence guarantees under KL assumptions. Key contributions include a practical algorithm with closed-form subproblem updates, a rigorous convergence analysis, and extensive experiments on synthetic and real datasets demonstrating improved accuracy over state-of-the-art symmetric nonnegative matrix factorization baselines. The work advances efficient, non-convex, non-smooth optimization for structured matrix decompositions and has potential impact on covariance estimation, spectral methods, and graph-based learning under non-negativity and sparsity constraints.

Abstract

Symmetric matrix decomposition is an active research area in machine learning. This paper focuses on exploiting the low-rank structure of non-negative and sparse symmetric matrices via the rectified linear unit (ReLU) activation function. We propose the ReLU-based nonlinear symmetric matrix decomposition (ReLU-NSMD) model, introduce an accelerated alternating partial Bregman (AAPB) method for its solution, and present the algorithm's convergence results. Our algorithm leverages the Bregman proximal gradient framework to overcome the challenge of estimating the global $L$-smooth constant in the classic proximal gradient algorithm. Numerical experiments on synthetic and real datasets validate the effectiveness of our model and algorithm.

An Efficient Alternating Algorithm for ReLU-based Symmetric Matrix Decomposition

TL;DR

The paper tackles low-rank symmetric matrix decomposition for non-negative, sparse data by formulating with . It introduces ReLU-NSMD and an accelerated alternating partial Bregman (AAPB) method that alternates between a closed-form -update and a -update solved via a Bregman proximal gradient step under the -smad condition, achieving convergence guarantees under KL assumptions. Key contributions include a practical algorithm with closed-form subproblem updates, a rigorous convergence analysis, and extensive experiments on synthetic and real datasets demonstrating improved accuracy over state-of-the-art symmetric nonnegative matrix factorization baselines. The work advances efficient, non-convex, non-smooth optimization for structured matrix decompositions and has potential impact on covariance estimation, spectral methods, and graph-based learning under non-negativity and sparsity constraints.

Abstract

Symmetric matrix decomposition is an active research area in machine learning. This paper focuses on exploiting the low-rank structure of non-negative and sparse symmetric matrices via the rectified linear unit (ReLU) activation function. We propose the ReLU-based nonlinear symmetric matrix decomposition (ReLU-NSMD) model, introduce an accelerated alternating partial Bregman (AAPB) method for its solution, and present the algorithm's convergence results. Our algorithm leverages the Bregman proximal gradient framework to overcome the challenge of estimating the global -smooth constant in the classic proximal gradient algorithm. Numerical experiments on synthetic and real datasets validate the effectiveness of our model and algorithm.

Paper Structure

This paper contains 10 sections, 3 theorems, 36 equations, 2 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Algorithm
  3. Details of Algorithm \ref{['NSMD-AAPB']}
  4. $W$-subproblem
  5. $U$-subproblem
  6. Convergence analysis
  7. Numerical experiments
  8. Sythetic datasets
  9. Real datasets
  10. Conclusion

Key Result

Proposition 1

In $k$-th iteration, let function $\psi:\mathbb{R}^{m\times r}\rightarrow\mathbb{R}$ be a kernel given by Then $F(W^{k+1},U)$ is $L$-smad relative to $\psi$ with $L\ge 1$.

Figures (2)

  • Figure 1: Numeric results for the synthetic dataset with different $m$, $\bar{r}$, $p$, and $r$ under Algorithm \ref{['NSMD-AAPB']} for different $\beta_{k}$.
  • Figure 2: Numeric results for four real datasets in Table \ref{['details_data']} under four compared algorithms.

Theorems & Definitions (7)

  • Example 1
  • Definition 1
  • Proposition 1
  • proof
  • Proposition 2
  • proof
  • Theorem 1