HOQRI: Higher-order QR Iteration for Low Multilinear Rank Approximation of Large and Sparse Tensors

TL;DR

This work introduces HOQRI, a QR-based higher-order iteration for Tucker-based LMLRA of large, sparse tensors, addressing memory bottlenecks and SVD dependence present in HOOI. It defines a memory-efficient TTMcTC kernel to update factor matrices without forming large intermediates and proves convergence to the set of stationary points within the product of Stiefel manifolds. The approach yields monotonic objective improvement and scalable per-iteration costs, validated by extensive experiments in MATLAB and C++ that outperform state-of-the-art baselines on synthetic and real datasets. Overall, HOQRI provides a practical, theoretically grounded framework for fast and memory-efficient tensor factorization in large-scale settings.

Abstract

We propose a new algorithm called higher-order QR iteration (HOQRI) for computing low multilinear rank approximation (LMLRA), also known as the Tucker decomposition, of large and sparse tensors. Compared to the celebrated higher-order orthogonal iterations (HOOI), HOQRI relies on a simple orthogonalization step in each iteration rather than a more sophisticated singular value decomposition step as in HOOI. More importantly, when dealing with extremely large and sparse data tensors, HOQRI completely eliminates the intermediate memory explosion by defining a new sparse tensor operation called TTMcTC (short for tensor times matrix chains times core). Furthermore, recognizing that the orthonormal constraints form a Cartesian product of Stiefel manifolds, we introduce the framework of manifold optimization and show that HOQRI guarantees convergence to the set of stationary points. Numerical experiments on synthetic and real data showcase the effectiveness of HOQRI.

Figures (12)

• Figure 1: Averaged per-iteration elapsed time of HOQRI comparing with Tensorlab tensorlab3.0, Tensor Toolbox tensor_toolbox, and S-HOT oh2017s on randomly generated sparse tensors. The target multilinear rank for the Tucker decomposition is fixed at $10\times10\times10$, and the dimension of the tensor increases from $100\times100\times100$ to $10^6\times10^6\times10^6$. The number of nonzeros of each tensor is $10\times$ its number of rows.
• Figure 2: Tucker decomposition of a 3-way tensor
• Figure 3: Convergence of HOQRI on 3-dimensional tensors versus the methods lmlra_hooi, lmlra_minf, lmlra_nls, and tucker_als with $I=600$.
• Figure 4: Convergence of HOQRI on 4-dimensional tensors versus the methods lmlra_hooi and tucker_als with $I=200$.
• Figure 5: Convergence of HOQRI on 5-dimensional tensors versus the methods lmlra_hooi and tucker_als with $I=80$.

Theorems & Definitions (9)

• Theorem 1
• proof
• Definition 1: Riemannian gradient
• Definition 2: Stationary point
• Theorem 2
• proof
• Lemma 1
• Theorem 3
• proof : Proof of Lemma \ref{['lmm:interlacing']}