Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Inertial Accelerated Stochastic Mirror Descent for Large-Scale Generalized Tensor CP Decomposition

Zehui Liu, Qingsong Wang, Chunfeng Cui, Yong Xia

TL;DR

The sublinear convergence rate for the subsequential sequence produced by the iTableSMD algorithm is demonstrated and it is shown that iTableSMD requires at most O(ε-2) iterations to attain an ε-2-stationary point and establish the global convergence of the sequence.

Abstract

The majority of classic tensor CP decomposition models are designed for squared loss, employing Euclidean distance as a local proximal term. However, the Euclidean distance is unsuitable for the generalized loss function applicable to various types of real-world data, such as integer and binary data. Consequently, algorithms developed under the squared loss are not easily adaptable to handle these generalized losses, partially due to the lack of the gradient Lipschitz continuity. This paper considers the generalized tensor CP decomposition. We use the Bregman distance as the proximal term and propose an inertial accelerated block randomized stochastic mirror descent algorithm (iTableSMD). Within a broader multi-block variance reduction and inertial acceleration framework, we demonstrate the sublinear convergence rate for the subsequential sequence produced by the iTableSMD algorithm. We further show that iTableSMD requires at most O(ε^{-2}) iterations in expectation to attain an ε-stationary point and establish the global convergence of the sequence. Numerical experiments on real datasets demonstrate that our proposed algorithm is efficient and achieve better performance than the existing state-of-the-art methods.

Inertial Accelerated Stochastic Mirror Descent for Large-Scale Generalized Tensor CP Decomposition

TL;DR

The sublinear convergence rate for the subsequential sequence produced by the iTableSMD algorithm is demonstrated and it is shown that iTableSMD requires at most O(ε-2) iterations to attain an ε-2-stationary point and establish the global convergence of the sequence.

Abstract

The majority of classic tensor CP decomposition models are designed for squared loss, employing Euclidean distance as a local proximal term. However, the Euclidean distance is unsuitable for the generalized loss function applicable to various types of real-world data, such as integer and binary data. Consequently, algorithms developed under the squared loss are not easily adaptable to handle these generalized losses, partially due to the lack of the gradient Lipschitz continuity. This paper considers the generalized tensor CP decomposition. We use the Bregman distance as the proximal term and propose an inertial accelerated block randomized stochastic mirror descent algorithm (iTableSMD). Within a broader multi-block variance reduction and inertial acceleration framework, we demonstrate the sublinear convergence rate for the subsequential sequence produced by the iTableSMD algorithm. We further show that iTableSMD requires at most O(ε^{-2}) iterations in expectation to attain an ε-stationary point and establish the global convergence of the sequence. Numerical experiments on real datasets demonstrate that our proposed algorithm is efficient and achieve better performance than the existing state-of-the-art methods.
Paper Structure (18 sections, 11 theorems, 68 equations, 5 figures, 2 tables, 1 algorithm)

This paper contains 18 sections, 11 theorems, 68 equations, 5 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Generalized CP decomposition
  4. Stochastic methods for GCP decomposition
  5. Stochastic mirror descent
  6. Inertial accelerated block-randomized SMD
  7. Convergence analysis
  8. Subsequential convergence analysis
  9. Global convergence analysis
  10. Numerical experiments
  11. Synthetic data experiments
  12. Gamma distribution
  13. Poisson distribution
  14. Bernoulli distribution
  15. Real data experiments
  16. ...and 3 more sections

Key Result

Lemma 1

Suppose Assumptionassume_01 is satisfied and $\tilde{\nabla}_{A_{n}}f$ with $n=1,2\dots,N$, is variance-reduced by Definition vr_definition. Let $\{A_{n}^{k}\}_{k>0}$ with $n\in\{1,\dots,N\}$ be the sequence generated by Algorithm iTableSMD. Then the following inequality holds for any $k>0$, Here, $\bar{\gamma}=\sqrt{2(V_{\Gamma}/\tau+V_{1})}$, $\alpha$ is the weakly convex parameter in Assumptio

Figures (5)

  • Figure 1: Numerical experiments for Gamma distribution on synthetic datasets.
  • Figure 2: Numerical experiments for Poisson distribution on synthetic datasets.
  • Figure 3: Numerical experiments for Bernoulli distribution on synthetic datasets.
  • Figure 4: Numerical experiments for Poisson distribution on Enron emails dataset.
  • Figure 5: Numerical experiments for Bernoulli distribution on the Flickr dataset.

Theorems & Definitions (24)

  • Definition 1
  • Remark 1
  • Definition 2
  • Definition 3
  • Definition 4
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Theorem 1
  • ...and 14 more