Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distributed-Memory Randomized Algorithms for Sparse Tensor CP Decomposition

Vivek Bharadwaj, Osman Asif Malik, Riley Murray, Aydin Buluç, James Demmel

TL;DR

The paper develops scalable distributed-memory randomized CP decomposition for billion-scale sparse tensors, introducing two algorithms, CP-ARLS-LEV and STS-CP, that leverage leverage-score sampling to accelerate ALS-like updates. It presents novel distributed sampling procedures, a communication-optimized MTTKRP schedule, and a tailored local tensor storage format, achieving substantial speedups (up to about 11x) over state-of-the-art SPLATT on real-world tensors. Experiments show that the methods maintain accuracy comparable to non-randomized ALS while delivering strong scaling on thousands of CPU cores, with tradeoffs between speed and precision controlled by the sample count $J$. The work significantly improves practical feasibility of large-scale sparse tensor CP decomposition, enabling faster insights from massive multidimensional data in domains such as social networks, text corpora, and knowledge graphs.

Abstract

Candecomp / PARAFAC (CP) decomposition, a generalization of the matrix singular value decomposition to higher-dimensional tensors, is a popular tool for analyzing multidimensional sparse data. On tensors with billions of nonzero entries, computing a CP decomposition is a computationally intensive task. We propose the first distributed-memory implementations of two randomized CP decomposition algorithms, CP-ARLS-LEV and STS-CP, that offer nearly an order-of-magnitude speedup at high decomposition ranks over well-tuned non-randomized decomposition packages. Both algorithms rely on leverage score sampling and enjoy strong theoretical guarantees, each with varying time and accuracy tradeoffs. We tailor the communication schedule for our random sampling algorithms, eliminating expensive reduction collectives and forcing communication costs to scale with the random sample count. Finally, we optimize the local storage format for our methods, switching between analogues of compressed sparse column and compressed sparse row formats. Experiments show that our methods are fast and scalable, producing 11x speedup over SPLATT by decomposing the billion-scale Reddit tensor on 512 CPU cores in under two minutes.

Distributed-Memory Randomized Algorithms for Sparse Tensor CP Decomposition

TL;DR

The paper develops scalable distributed-memory randomized CP decomposition for billion-scale sparse tensors, introducing two algorithms, CP-ARLS-LEV and STS-CP, that leverage leverage-score sampling to accelerate ALS-like updates. It presents novel distributed sampling procedures, a communication-optimized MTTKRP schedule, and a tailored local tensor storage format, achieving substantial speedups (up to about 11x) over state-of-the-art SPLATT on real-world tensors. Experiments show that the methods maintain accuracy comparable to non-randomized ALS while delivering strong scaling on thousands of CPU cores, with tradeoffs between speed and precision controlled by the sample count . The work significantly improves practical feasibility of large-scale sparse tensor CP decomposition, enabling faster insights from massive multidimensional data in domains such as social networks, text corpora, and knowledge graphs.

Abstract

Candecomp / PARAFAC (CP) decomposition, a generalization of the matrix singular value decomposition to higher-dimensional tensors, is a popular tool for analyzing multidimensional sparse data. On tensors with billions of nonzero entries, computing a CP decomposition is a computationally intensive task. We propose the first distributed-memory implementations of two randomized CP decomposition algorithms, CP-ARLS-LEV and STS-CP, that offer nearly an order-of-magnitude speedup at high decomposition ranks over well-tuned non-randomized decomposition packages. Both algorithms rely on leverage score sampling and enjoy strong theoretical guarantees, each with varying time and accuracy tradeoffs. We tailor the communication schedule for our random sampling algorithms, eliminating expensive reduction collectives and forcing communication costs to scale with the random sample count. Finally, we optimize the local storage format for our methods, switching between analogues of compressed sparse column and compressed sparse row formats. Experiments show that our methods are fast and scalable, producing 11x speedup over SPLATT by decomposing the billion-scale Reddit tensor on 512 CPU cores in under two minutes.
Paper Structure (29 sections, 1 theorem, 21 equations, 13 figures, 5 tables, 5 algorithms)

This paper contains 29 sections, 1 theorem, 21 equations, 13 figures, 5 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Motivation
  3. Our Contributions
  4. Novel Distributed-Memory Sampling Procedures
  5. Communication-Optimized MTTKRP
  6. Local Tensor Storage Format
  7. Notation and Preliminaries
  8. Non-Randomized ALS CP Decomposition
  9. Randomized Leverage Score Sampling
  10. Related Work
  11. High-Performance ALS CP Decomposition
  12. Alternate Sketching Algorithms and Tensor Decomposition Methods
  13. Distributed-Randomized CP Decomposition
  14. New Distributed Sampling Strategies
  15. A Randomization-Tailored MTTKRP Schedule
  16. ...and 14 more sections

Key Result

Theorem 2.1

Let $S \in \mathbb{R}^{J \times I}$ be a sampling matrix for $A \in \mathbb{R}^{I \times R}$ where each row $i$ is sampled i.i.d. with probability $p_i$. Let $\beta = \min_{i \in \left[ I \right]} (p_i R / \ell_i)$. For a constant $C$ and any $\varepsilon, \delta \in (0, 1)$, let the sample count be Letting $\tilde{X} = \textrm{argmin}_{\tilde{X}} \left\lVert SA \tilde{X} - SB\right\rVert_F$, we h

Figures (13)

  • Figure 1: A subset of entries from the 3D Amazon Review sparse tensor smith_frostt_2017 and its illustrated CP decomposition. The sparse tensor is approximated by a sum of 3D outer products of vectors, which are columns of factor matrices $U_1, U_2$, and $U_3$. Outer products are scaled by elements of $\sigma$.
  • Figure 2: Running maximum accuracy over time for SPLATT, a state-of-the-art distributed CP decomposition software package, and our randomized algorithms on the Reddit tensor, target rank $R=100$, on 512 CPU cores. Curves are averages of 5 trials, 80 ALS rounds.
  • Figure 3: Top: the linear least-squares problem to optimize factor matrix $U_2$ during the ALS algorithm for a 3D tensor (column dimension of $\textrm{mat}(\mathcal{T}, 2)$ not to scale). Middle: the exact solution to the problem using the Matricized Tensor Times Khatri-Rao Product (MTTKRP). Shaded columns of $\textrm{mat}(\mathcal{T}, 2)$ and rows of $(U_3 \odot U_1)$ are selected by our random sampling algorithm. Bottom: the downsampled linear least-squares problem after applying random sampling matrix $S$.
  • Figure 4: Initial data distribution and downsampled MTTKRP data movement for a 3D tensor, $P=8$ processors. Rectangles along each side of the tensor illustrate factor matrices corresponding to each mode, divided by block rows among processors. Each black circle denotes the processor owning a block of a matrix or tensor; multiple circles on an object indicate replication of a piece of data. Colors / shading indicate communication collectives.
  • Figure 5: Example random walk in STS-CP to draw a single sample index from matrix $U_1$, distributed to $P=4$ processors. Annotations on the tree (left) indicate processors that share data for each node. The schedule to the right indicates the processor that owns the sample at each stage of the random walk. The sample begins randomly at $p_2$, then branches left to $p_3$ ($p_4$ shares node data and could also have been selected), involving communication of a vector corresponding to the sample from $p_2$ to $p_3$. The sample remains at $p_3$ for the remainder of the walk.
  • ...and 8 more figures

Theorems & Definitions (1)

  • Theorem 2.1: Larsen and Kolda larsen_practical_2022