Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Riemannian preconditioned coordinate descent for low multi-linear rank approximation

Mohammad Hamed, Reshad Hosseini

TL;DR

This paper presents a memory efficient, first-order method for low multi-linear rank approximation of high-order, high-dimensional tensors and uses a Riemmanian coordinate descent method for solving the problem, and provides a global convergence analysis matching that of the coordinate descent method in the Euclidean setting.

Abstract

This paper presents a memory efficient, first-order method for low multi-linear rank approximation of high-order, high-dimensional tensors. In our method, we exploit the second-order information of the cost function and the constraints to suggest a new Riemannian metric on the Grassmann manifold. We use a Riemmanian coordinate descent method for solving the problem, and also provide a global convergence analysis matching that of the coordinate descent method in the Euclidean setting. We also show that each step of our method with the unit step-size is actually a step of the orthogonal iteration algorithm. Experimental results show the computational advantage of our method for high-dimensional tensors.

Riemannian preconditioned coordinate descent for low multi-linear rank approximation

TL;DR

This paper presents a memory efficient, first-order method for low multi-linear rank approximation of high-order, high-dimensional tensors and uses a Riemmanian coordinate descent method for solving the problem, and provides a global convergence analysis matching that of the coordinate descent method in the Euclidean setting.

Abstract

This paper presents a memory efficient, first-order method for low multi-linear rank approximation of high-order, high-dimensional tensors. In our method, we exploit the second-order information of the cost function and the constraints to suggest a new Riemannian metric on the Grassmann manifold. We use a Riemmanian coordinate descent method for solving the problem, and also provide a global convergence analysis matching that of the coordinate descent method in the Euclidean setting. We also show that each step of our method with the unit step-size is actually a step of the orthogonal iteration algorithm. Experimental results show the computational advantage of our method for high-dimensional tensors.

Paper Structure

This paper contains 12 sections, 9 theorems, 64 equations, 4 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Preliminaries and background
  4. Definitions
  5. Riemannian preconditioning
  6. Problem statement
  7. Riemannian Preconditioned Coordinate Descent
  8. Convergence analysis
  9. Experimental Results
  10. Synthetic Data
  11. Real Data
  12. Conclusion

Key Result

Theorem 8

\newlabel2.90 Consider an equivalence relation $\sim$ in $\mathcal{M}$. Assume that both $\mathcal{M}$ and $\mathcal{M}/$$\sim$ have the structure of a Riemannian manifold and a function $f: \mathcal{M}\rightarrow \mathbb{R}$ is a smooth function with isolated minima on the quotient manifold. Assu where $\mathcal{V}_{x^*}$ is the vertical space, and $\mathcal{H}_{x^*}$ is the horizontal space (th

Figures (4)

  • Figure 1: Convergence of Riemannian coordinate descent with the Euclidean metric for decomposing a random tensor having a low multi-linear rank. The best attainable relative error is zero, and it is clear that the coordinate descent method has convergence problems in the Euclidean space.
  • Figure 1: Compression of Yale face database (1th row) with HOOI (2th row) and RPCD+ (3th row)
  • Figure 2: Convergence behavior of different methods for the real datasets. Y-axis is the difference between the relative error at each iteration and the best achieved relative error.
  • Figure 3: Convergence behavior for decomposing the Brainq dataset using different random initializations.

Theorems & Definitions (25)

  • Definition 1: Tensor
  • Definition 2: Matricization (unfolding)
  • Definition 3: Multi-linear rank
  • Definition 4: $i$-mode product
  • Definition 5: Tensor norm
  • Definition 6: Stiefel manifold $St(n,r)$
  • Definition 7: Grassmann manifold $Gr(n,r)$
  • Theorem 8: Theorem 3.1 in mishra2016riemannian
  • Definition 1: Vector transport absil2009optimization
  • Definition 2: Radially Lipschitz continuously differentiable function
  • ...and 15 more