Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Momentum Accelerated Algorithm for ReLU-based Nonlinear Matrix Decomposition

Qingsong Wang, Chunfeng Cui, Deren Han

TL;DR

This work proposes a Tikhonov regularized ReLU-NMD model, referred to as ReLU-NMD-T, and introduces a momentum accelerated algorithm for handling the ReLU-NMD-T model, which incorporates both positive and negative momentum parameters in its algorithm.

Abstract

Recently, there has been a growing interest in the exploration of Nonlinear Matrix Decomposition (NMD) due to its close ties with neural networks. NMD aims to find a low-rank matrix from a sparse nonnegative matrix with a per-element nonlinear function. A typical choice is the Rectified Linear Unit (ReLU) activation function. To address over-fitting in the existing ReLU-based NMD model (ReLU-NMD), we propose a Tikhonov regularized ReLU-NMD model, referred to as ReLU-NMD-T. Subsequently, we introduce a momentum accelerated algorithm for handling the ReLU-NMD-T model. A distinctive feature, setting our work apart from most existing studies, is the incorporation of both positive and negative momentum parameters in our algorithm. Our numerical experiments on real-world datasets show the effectiveness of the proposed model and algorithm. Moreover, the code is available at https://github.com/nothing2wang/NMD-TM.

A Momentum Accelerated Algorithm for ReLU-based Nonlinear Matrix Decomposition

TL;DR

This work proposes a Tikhonov regularized ReLU-NMD model, referred to as ReLU-NMD-T, and introduces a momentum accelerated algorithm for handling the ReLU-NMD-T model, which incorporates both positive and negative momentum parameters in its algorithm.

Abstract

Recently, there has been a growing interest in the exploration of Nonlinear Matrix Decomposition (NMD) due to its close ties with neural networks. NMD aims to find a low-rank matrix from a sparse nonnegative matrix with a per-element nonlinear function. A typical choice is the Rectified Linear Unit (ReLU) activation function. To address over-fitting in the existing ReLU-based NMD model (ReLU-NMD), we propose a Tikhonov regularized ReLU-NMD model, referred to as ReLU-NMD-T. Subsequently, we introduce a momentum accelerated algorithm for handling the ReLU-NMD-T model. A distinctive feature, setting our work apart from most existing studies, is the incorporation of both positive and negative momentum parameters in our algorithm. Our numerical experiments on real-world datasets show the effectiveness of the proposed model and algorithm. Moreover, the code is available at https://github.com/nothing2wang/NMD-TM.
Paper Structure (7 sections, 8 equations, 7 figures, 1 algorithm)

This paper contains 7 sections, 8 equations, 7 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Algorithm
  3. Numerical experiments
  4. Setting of $\beta$
  5. MNIST dataset
  6. Compression of sparse NMF basis
  7. Conclusion and Future Work

Figures (7)

  • Figure 1: Illustration of the positive momentum acceleration of $W$ with $\alpha\in(0,1)$ (left) and negative momentum acceleration of $U$ with $\beta\in(0,1)$ (right).
  • Figure 2: Numerical experiments of MNIST with $5000$ images ($500$ images of each digit) under different $\beta$ by the NMD-TM algorithm. Left: $r=30$. Right: $r=45$.
  • Figure 3: Numerical experiments of MNIST with $5000$ images ($500$ images of each digit). Left: $r=30$. Right: $r=50$.
  • Figure 4: Numerical experiments with different sample numbers of the MNIST data set with rank $r=40$.
  • Figure 5: Final relative error on $m=50000$ images from MNIST data set, after 20 seconds and average iteration time for increasing value of the rank $r$. Left: Relative error. Right: Time (seconds).
  • ...and 2 more figures

Theorems & Definitions (2)

  • Example 1
  • Remark 1