Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient approximations of matrix multiplication using truncated decompositions

Suvendu Kar, Hariprasad M., Sai Gowri J. N., Murugesan Venkatapathi

TL;DR

The paper develops practical, first-order methods to approximate products of large dense matrices by decomposing each factor into a small set of dominant components plus a residue, using truncated SVD, cycle, circulant, and Fourier-based decompositions. By combining dominant parts with cross-terms involving residues, and neglecting the residue-residue term, it achieves $O(n^2 \log n)$ arithmetic with relative errors near 1% for moderate tolerances, and provides a priori error estimates. The authors also present a thorough error analysis under different random-spectral models, along with extensive numerical results showing complementary strengths of SVD-based and circulant-based approaches, and situating sparsification within a first-order framework. They further compare run-time performance against randomized outer-product methods, demonstrating potential order-of-magnitude speedups on large matrices in practical settings. The work offers practical guidance for selecting decompositions, estimating errors, and implementing FFT-based acceleration for large-scale matrix multiplication tasks.

Abstract

We exploit the truncated singular value decomposition and the recently proposed circulant decomposition for an efficient first-order approximation of the multiplication of large dense matrices. A decomposition of each matrix into a sum of a sparse matrix with relatively few dominant entries and a dense residue can also use the above approach, and we present methods for multiplication using a Fourier decomposition and a cycle decomposition-based sparsifications. The proposed methods scale as $\mathcal{O}(n^2 \log n)$ in arithmetic operations for $n \times n$ matrices for usable tolerances in relative error $\sim$ 1\%. Note that different decompositions for the two matrices $A$ and $B$ in the product $AB$ are also possible in this approach, using efficient a priori evaluations for suitability, to improve further on the error tolerances demonstrated here.

Efficient approximations of matrix multiplication using truncated decompositions

TL;DR

The paper develops practical, first-order methods to approximate products of large dense matrices by decomposing each factor into a small set of dominant components plus a residue, using truncated SVD, cycle, circulant, and Fourier-based decompositions. By combining dominant parts with cross-terms involving residues, and neglecting the residue-residue term, it achieves arithmetic with relative errors near 1% for moderate tolerances, and provides a priori error estimates. The authors also present a thorough error analysis under different random-spectral models, along with extensive numerical results showing complementary strengths of SVD-based and circulant-based approaches, and situating sparsification within a first-order framework. They further compare run-time performance against randomized outer-product methods, demonstrating potential order-of-magnitude speedups on large matrices in practical settings. The work offers practical guidance for selecting decompositions, estimating errors, and implementing FFT-based acceleration for large-scale matrix multiplication tasks.

Abstract

We exploit the truncated singular value decomposition and the recently proposed circulant decomposition for an efficient first-order approximation of the multiplication of large dense matrices. A decomposition of each matrix into a sum of a sparse matrix with relatively few dominant entries and a dense residue can also use the above approach, and we present methods for multiplication using a Fourier decomposition and a cycle decomposition-based sparsifications. The proposed methods scale as in arithmetic operations for matrices for usable tolerances in relative error 1\%. Note that different decompositions for the two matrices and in the product are also possible in this approach, using efficient a priori evaluations for suitability, to improve further on the error tolerances demonstrated here.
Paper Structure (30 sections, 12 theorems, 58 equations, 34 figures, 4 tables, 8 algorithms)

This paper contains 30 sections, 12 theorems, 58 equations, 34 figures, 4 tables, 8 algorithms.

Table of Contents

  1. Introduction
  2. Notations used
  3. Relevant Decompositions
  4. Singular Value Decomposition
  5. Decomposition into cycles
  6. Circulant Decomposition
  7. Fourier series based decompositions
  8. Methods
  9. Truncated decompositions and a first-order approximation
  10. Based on Singular Value Decomposition
  11. Based on Circulant Decomposition
  12. Evaluating circulant components $R_k$
  13. Multiplication using circulant components
  14. Based on Fourier-domain sparsification
  15. Other theoretical aspects in practical implementation:
  16. ...and 15 more sections

Key Result

Theorem 1.1

\newlabelth:circulant_decomposition0 Any $n \times n$ matrix A is given by a sum of $n$ circulant matrices $R_k$ with phase factors given by roots of unity. $R_kD^k$ are mutually orthogonal w.r.t. a Frobenius inner product $\odot$, and $A \odot R_kD^k = \lVert R_k\rVert_F^2$. Note that $A \odot A = \sum\limits_{i,j} \overline{A(i,j)}A(i,j) = \lVert A\rVert_F^2$.

Figures (34)

  • Figure 1: Size of matrix vs. mean relative error in multiplying two Type-1 matrices. $5\lceil \log n \rceil$ components of the SVD were used. A priori estimates using Theorem 3.6 ignoring its $\mathcal{O}(\frac{1}{\sqrt{n}})$ second term, and posterior estimates using Corollary 3.7 are also plotted above. While the former is smaller than the actual errors for these moderate values of $n$, the latter provides estimates that overlap with the observed relative error.
  • Figure 2: Size vs. mean relative error in multiplying two Type-2 matrices. $5\lceil \log n \rceil$ components of the SVD were used. A priori estimates using Theorem 3.6 ignoring its $\mathcal{O}(\frac{1}{\sqrt{n}})$ second term, and posterior estimates using Corollary 3.7 are also plotted above. While the former is smaller than the actual errors for these moderate values of $n$, the latter provides estimates that overlap with the observed relative error.
  • Figure 3: Size of matrix vs. mean relative error in multiplying a general and a Toeplitz matrix with randomly generated entries from $\mathcal{U}$(0,1). $5\lceil \log n \rceil$ components of the CD were used. A priori estimates using Theorem 3.6 ignoring its $\mathcal{O}(\frac{1}{\sqrt{n}})$ second term, and posterior estimates using Corollary 3.7 are also plotted above. Both provide estimates that overlap with the observed relative error.
  • Figure 4: Size of matrix vs. mean relative error in multiplying two random matrices with entries from a $\mathcal{U}$(0,1). $5\lceil \log n \rceil$ components of the CD were used. A priori estimates using Theorem 3.6 ignoring its $\mathcal{O}(\frac{1}{\sqrt{n}})$ second term, and posterior estimates using Corollary 3.7 are also plotted above. Both provide estimates that overlap with the observed relative error.
  • Figure 5: Size of matrix vs. mean relative error in multiplying a general and a Toeplitz matrix with randomly generated entries from a $\mathcal{U}$(0,1) distribution. $5\lceil \log n \rceil$ entries for every row/column were used in the FFT-based sparsification.
  • ...and 29 more figures

Theorems & Definitions (23)

  • Theorem 1.1
  • Proof 1
  • Theorem 1.2
  • Proof 2
  • Theorem 2.1
  • Proof 3
  • Lemma 3.1
  • Proof 4
  • Theorem 3.2
  • Corollary 3.3
  • ...and 13 more