Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Numerical Matrix Decomposition

Jun Lu

TL;DR

Numerical Matrix Decomposition surveys core matrix factorization methods (LU, PLU, LDU, Cholesky, QR, SVD, and related forms) and their algorithmic and practical implications. It connects classical Gaussian elimination to modern decomposition techniques, analyzes pivoting strategies, block and rank-revealing approaches, and derives complexity bounds such as ~ (2/3) n^3 flops for LU factorization. The text emphasizes the Schur complement, matrix inversion lemmas (Woodbury, Sherman–Morrison), and bandwidth considerations, illustrating how these tools enable efficient solutions to linear systems and matrix inverses, as well as low-rank and block-structured decompositions. Through detailed sections, it bridges theoretical linear algebra with numerical algorithms, highlighting applications in solving linear systems, inverses, and decomposition-based representations with clear, rigorous proofs. The work thus provides a comprehensive, decomposition-centered foundation for both classical numerical linear algebra and contemporary matrix-factorization techniques used in optimization, data analysis, and machine learning.

Abstract

In 1954, Alston S. Householder published \textit{Principles of Numerical Analysis}, one of the first modern treatments on matrix decomposition that favored a (block) LU decomposition-the factorization of a matrix into the product of lower and upper triangular matrices. And now, matrix decomposition has become a core technology in machine learning, largely due to the development of the back propagation algorithm in fitting a neural network. The sole aim of this survey is to give a self-contained introduction to concepts and mathematical tools in numerical linear algebra and matrix analysis in order to seamlessly introduce matrix decomposition techniques and their applications in subsequent sections. However, we clearly realize our inability to cover all the useful and interesting results concerning matrix decomposition and given the paucity of scope to present this discussion, e.g., the separated analysis of the Euclidean space, Hermitian space, Hilbert space, and things in the complex domain. We refer the reader to literature in the field of linear algebra for a more detailed introduction to the related fields. Keywords: Existence and computing of matrix decompositions, Floating point operations (flops), Low-rank approximation, Pivot, LU/PLU decomposition, CR/CUR/Skeleton decomposition, Coordinate transformation, ULV/URV decomposition, Rank decomposition, Rank revealing decomposition, Update/downdate, Tensor decomposition.

Numerical Matrix Decomposition

TL;DR

Numerical Matrix Decomposition surveys core matrix factorization methods (LU, PLU, LDU, Cholesky, QR, SVD, and related forms) and their algorithmic and practical implications. It connects classical Gaussian elimination to modern decomposition techniques, analyzes pivoting strategies, block and rank-revealing approaches, and derives complexity bounds such as ~ (2/3) n^3 flops for LU factorization. The text emphasizes the Schur complement, matrix inversion lemmas (Woodbury, Sherman–Morrison), and bandwidth considerations, illustrating how these tools enable efficient solutions to linear systems and matrix inverses, as well as low-rank and block-structured decompositions. Through detailed sections, it bridges theoretical linear algebra with numerical algorithms, highlighting applications in solving linear systems, inverses, and decomposition-based representations with clear, rigorous proofs. The work thus provides a comprehensive, decomposition-centered foundation for both classical numerical linear algebra and contemporary matrix-factorization techniques used in optimization, data analysis, and machine learning.

Abstract

In 1954, Alston S. Householder published \textit{Principles of Numerical Analysis}, one of the first modern treatments on matrix decomposition that favored a (block) LU decomposition-the factorization of a matrix into the product of lower and upper triangular matrices. And now, matrix decomposition has become a core technology in machine learning, largely due to the development of the back propagation algorithm in fitting a neural network. The sole aim of this survey is to give a self-contained introduction to concepts and mathematical tools in numerical linear algebra and matrix analysis in order to seamlessly introduce matrix decomposition techniques and their applications in subsequent sections. However, we clearly realize our inability to cover all the useful and interesting results concerning matrix decomposition and given the paucity of scope to present this discussion, e.g., the separated analysis of the Euclidean space, Hermitian space, Hilbert space, and things in the complex domain. We refer the reader to literature in the field of linear algebra for a more detailed introduction to the related fields. Keywords: Existence and computing of matrix decompositions, Floating point operations (flops), Low-rank approximation, Pivot, LU/PLU decomposition, CR/CUR/Skeleton decomposition, Coordinate transformation, ULV/URV decomposition, Rank decomposition, Rank revealing decomposition, Update/downdate, Tensor decomposition.

Paper Structure

This paper contains 989 sections, 1901 equations, 92 figures, 7 tables, 81 algorithms.

Table of Contents

  1. Introduction and Background
  2. Notation and preliminaries.
  3. Linear Algebra
  4. The vector space $\mathbb{R}^n$ and matrix space $\mathbb{R}^{m\times n}$.
  5. Lemma 1.16: (Premultiplying with Nonsingular)
  6. Subspace intersection.
  7. Complementary nullities.
  8. Fredholm alternative theorem.
  9. Row perspective.
  10. Column perspective.
  11. Reversal matrix.
  12. Generalized permutation matrices.
  13. Row perspective.
  14. Column perspective.
  15. Observation.
  16. ...and 974 more sections

Figures (92)

  • Figure 1: Matrix Decomposition World Map.
  • Figure 2: Matrix Decomposition World Map Under Conditions.
  • Figure 3: Plot for function $f(x, y) = \sqrt{x^2+y^2}$, in which case any directional derivative for the direction $\bm{d}=(a,b)$ with $a\neq 0$ and $b\neq 0$ at point $(0,0)$ exists. However, the partial derivatives at this point do not exist.
  • Figure 4: Loss surfaces for different quadratic forms, providing the surface plots and contour plots (blue=low, yellow=high), where the upper graphs are the surface plots, and the lower ones are their projection (i.e., contours).
  • Figure 5: Demonstration of different factorizations for a positive definite matrix $\bm{A}$.
  • ...and 87 more figures

Theorems & Definitions (166)

  • Definition 1.1: Matlab Notation
  • Definition 1.2: Nonnegative Orthant, Positive Orthant, and Unit-Simplex
  • Definition 1.3: Eigenvalue, Eigenvector
  • Definition 1.4: Spectrum and Spectral Radius
  • Definition 1.6: Subspace
  • Definition 1.7: Span
  • Definition 1.8: Linearly Independent
  • Definition 1.9: Basis and Dimension
  • Definition 1.10: Column Space (Range)
  • Definition 1.11: Null Space (Nullspace, Kernel)
  • ...and 156 more