Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fast Updating Truncated SVD for Representation Learning with Sparse Matrices

Haoran Deng, Yang Yang, Jiahe Li, Cheng Chen, Weihao Jiang, Shiliang Pu

TL;DR

This work introduces a novel method for dynamically approximating the truncated SVD of a sparse and temporally evolving matrix that leverages sparsity in the orthogonalization process of augmented matrices and utilizes an extended decomposition to independently store projections in the column space of singular vectors.

Abstract

Updating a truncated Singular Value Decomposition (SVD) is crucial in representation learning, especially when dealing with large-scale data matrices that continuously evolve in practical scenarios. Aligning SVD-based models with fast-paced updates becomes increasingly important. Existing methods for updating truncated SVDs employ Rayleigh-Ritz projection procedures, where projection matrices are augmented based on original singular vectors. However, these methods suffer from inefficiency due to the densification of the update matrix and the application of the projection to all singular vectors. To address these limitations, we introduce a novel method for dynamically approximating the truncated SVD of a sparse and temporally evolving matrix. Our approach leverages sparsity in the orthogonalization process of augmented matrices and utilizes an extended decomposition to independently store projections in the column space of singular vectors. Numerical experiments demonstrate a remarkable efficiency improvement of an order of magnitude compared to previous methods. Remarkably, this improvement is achieved while maintaining a comparable precision to existing approaches.

Fast Updating Truncated SVD for Representation Learning with Sparse Matrices

TL;DR

This work introduces a novel method for dynamically approximating the truncated SVD of a sparse and temporally evolving matrix that leverages sparsity in the orthogonalization process of augmented matrices and utilizes an extended decomposition to independently store projections in the column space of singular vectors.

Abstract

Updating a truncated Singular Value Decomposition (SVD) is crucial in representation learning, especially when dealing with large-scale data matrices that continuously evolve in practical scenarios. Aligning SVD-based models with fast-paced updates becomes increasingly important. Existing methods for updating truncated SVDs employ Rayleigh-Ritz projection procedures, where projection matrices are augmented based on original singular vectors. However, these methods suffer from inefficiency due to the densification of the update matrix and the application of the projection to all singular vectors. To address these limitations, we introduce a novel method for dynamically approximating the truncated SVD of a sparse and temporally evolving matrix. Our approach leverages sparsity in the orthogonalization process of augmented matrices and utilizes an extended decomposition to independently store projections in the column space of singular vectors. Numerical experiments demonstrate a remarkable efficiency improvement of an order of magnitude compared to previous methods. Remarkably, this improvement is achieved while maintaining a comparable precision to existing approaches.
Paper Structure (26 sections, 5 theorems, 8 equations, 4 figures, 9 tables, 11 algorithms)

This paper contains 26 sections, 5 theorems, 8 equations, 4 figures, 9 tables, 11 algorithms.

Table of Contents

  1. Introduction
  2. Background and Notations
  3. Related Work
  4. Methodology
  5. Faster Orthogonalization of Augmented Matrix
  6. An Extended Decomposition to Reducing Complexity
  7. Main Result
  8. Time complexity comparing to previous methods
  9. Numerical Experiment
  10. Experimental Description
  11. Efficiency Study
  12. Varying $k$ and $\phi$
  13. Conclusion
  14. Reproducibility
  15. Notations
  16. ...and 11 more sections

Key Result

Lemma 1

For an orthonormal matrix ${\bm{U}}_k \in \mathbb{R}^{m\times k}$ with ${\bm{U}}_k^\top {\bm{U}}_k = {\bm{I}}$, turning $({\bm{I}}-{\bm{U}}_k{\bm{U}}_k^\top){\bm{b}}$ with a sparse vector ${\bm{b}} \in \mathbb{R}^{m}$ into SV-LCOV can be done in time complexity of $O(k\cdot nnz({\bm{b}}))$.

Figures (4)

  • Figure 1: Computational efficiency of adjacency matrix w.r.t. for $k$ values of $16,32,64,128,256$
  • Figure 2: Computational efficiency of adjacency matrix w.r.t. for $\phi$ values of $10^1, 10^2, 10^3, 10^4$
  • Figure 3: Runtime of each step on the Slashdot dataset.
  • Figure 4: Computational efficiency on synthetic matrices

Theorems & Definitions (11)

  • Definition 1: SV-LCOV
  • Lemma 1
  • Lemma 2
  • Lemma 3: Isometry of SV-LCOV
  • Lemma 4
  • Definition 2
  • Theorem 1: Main result
  • proof
  • proof
  • proof
  • ...and 1 more