Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Faster Algorithms for Schatten-p Low Rank Approximation

Praneeth Kacham, David P. Woodruff

TL;DR

This work advances Schatten-$p$ low rank approximation by deriving faster iterative algorithms that exploit fast rectangular matrix multiplication and block-size tradeoffs, and by combining these with Li and Woodruff's sketching-based LW20 approach to broaden the regime where near-linear-time, high-precision LRA is attainable. It also provides the first stability analysis for a FP-implementable variant of Block Krylov/LazySVD techniques, showing that a modified LazySVD can deliver $(1+\\varepsilon)$-approximate Schatten-$p$ LRA on machines with polylogarithmic precision when using $O((k/\\sqrt{\\varepsilon})\\operatorname{poly}(\\log n))$ matvecs. The results identify concrete time scales, e.g., $\\tilde{O}(\\sqrt{p} n^{2+\\eta})$ under favorable rank conditions and a combined bound of $\\tilde{O}((n^{1-2/p})^{2+\\eta+(1-\\alpha)\\beta/(1+2\\beta)}\\operatorname{poly}(1/\\varepsilon) + n^2)$, signaling practical pathways for fast, stable Schatten-$p$ LRA in dense regimes. Collectively, the paper argues for a pragmatic hybrid pipeline that blends sketching, fast Krylov-based subspace methods, and FP-stable least-squares steps to balance speed and numerical reliability in large-scale low-rank approximation tasks.

Abstract

We study algorithms for the Schatten-$p$ Low Rank Approximation (LRA) problem. First, we show that by using fast rectangular matrix multiplication algorithms and different block sizes, we can improve the running time of the algorithms in the recent work of Bakshi, Clarkson and Woodruff (STOC 2022). We then show that by carefully combining our new algorithm with the algorithm of Li and Woodruff (ICML 2020), we can obtain even faster algorithms for Schatten-$p$ LRA. While the block-based algorithms are fast in the real number model, we do not have a stability analysis which shows that the algorithms work when implemented on a machine with polylogarithmic bits of precision. We show that the LazySVD algorithm of Allen-Zhu and Li (NeurIPS 2016) can be implemented on a floating point machine with only logarithmic, in the input parameters, bits of precision. As far as we are aware, this is the first stability analysis of any algorithm using $O((k/\sqrt{\varepsilon})\text{poly}(\log n))$ matrix-vector products with the matrix $A$ to output a $1+\varepsilon$ approximate solution for the rank-$k$ Schatten-$p$ LRA problem.

Faster Algorithms for Schatten-p Low Rank Approximation

TL;DR

This work advances Schatten- low rank approximation by deriving faster iterative algorithms that exploit fast rectangular matrix multiplication and block-size tradeoffs, and by combining these with Li and Woodruff's sketching-based LW20 approach to broaden the regime where near-linear-time, high-precision LRA is attainable. It also provides the first stability analysis for a FP-implementable variant of Block Krylov/LazySVD techniques, showing that a modified LazySVD can deliver -approximate Schatten- LRA on machines with polylogarithmic precision when using matvecs. The results identify concrete time scales, e.g., under favorable rank conditions and a combined bound of , signaling practical pathways for fast, stable Schatten- LRA in dense regimes. Collectively, the paper argues for a pragmatic hybrid pipeline that blends sketching, fast Krylov-based subspace methods, and FP-stable least-squares steps to balance speed and numerical reliability in large-scale low-rank approximation tasks.

Abstract

We study algorithms for the Schatten- Low Rank Approximation (LRA) problem. First, we show that by using fast rectangular matrix multiplication algorithms and different block sizes, we can improve the running time of the algorithms in the recent work of Bakshi, Clarkson and Woodruff (STOC 2022). We then show that by carefully combining our new algorithm with the algorithm of Li and Woodruff (ICML 2020), we can obtain even faster algorithms for Schatten- LRA. While the block-based algorithms are fast in the real number model, we do not have a stability analysis which shows that the algorithms work when implemented on a machine with polylogarithmic bits of precision. We show that the LazySVD algorithm of Allen-Zhu and Li (NeurIPS 2016) can be implemented on a floating point machine with only logarithmic, in the input parameters, bits of precision. As far as we are aware, this is the first stability analysis of any algorithm using matrix-vector products with the matrix to output a approximate solution for the rank- Schatten- LRA problem.
Paper Structure (23 sections, 15 theorems, 61 equations, 1 figure, 2 tables, 5 algorithms)

This paper contains 23 sections, 15 theorems, 61 equations, 1 figure, 2 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Our Results
  3. Implications to Practice
  4. Preliminaries
  5. Notation
  6. Fast Rectangular Matrix Multiplication
  7. Schatten-p LRA using Fast Matrix Multiplication
  8. Block Krylov Iteration Algorithm
  9. Main Theorem
  10. Comparison with the Algorithm of Li and Woodruff LW20
  11. Further Improving the running time of LW20 using our algorithm
  12. Stability of LazySVD
  13. Finite Precision Preliminaries
  14. Stability Analysis
  15. Time Complexity of SVD in the Real RAM model
  16. ...and 8 more sections

Key Result

Theorem 1.1

Given an $n\times n$ matrix $A$, a rank parameter $k$ and an accuracy parameter $\varepsilon$, there is an algorithm that outputs a rank-$k$ orthonormal matrix $W$ that with probability $\ge 0.9$ satisfies, $\|A(I-WW^{\top})\|_{S_p} \le (1+O(\varepsilon))\|A - A_k\|_{S_p}.$ If $k \le \varepsilon \cd

Figures (1)

  • Figure 1: Color Map of $(j/i)/(t_j/t_i)$

Theorems & Definitions (23)

  • Theorem 1.1: Informal, Theorem \ref{['thm:final-theorem-our-algorithm']}
  • Theorem 1.2: Informal, Theorem \ref{['thm:combination']}
  • Theorem 1.3: Informal, Theorem \ref{['thm:lazysvd-stability']}
  • Theorem 3.1
  • Theorem 3.2
  • Lemma 4.1
  • proof
  • Lemma 4.2
  • proof
  • Theorem 4.3
  • ...and 13 more