Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Small singular values can increase in lower precision

Christos Boutsikas, Petros Drineas, Ilse C. F. Ipsen

TL;DR

The paper addresses how downcasting a tall, full-column-rank matrix $\boldsymbol{A}$ to lower arithmetic precision can increase its smallest singular values. It models precision demotion as a normwise perturbation $\boldsymbol{E}$ and develops deterministic lower bounds for the smallest singular value, first for a single value via Theorems $t_{dp4}$/$t_{dp5}$ and then for a cluster via Theorems $t_{dp6}$/$t_{dp7}$ (and the related $t_{det4}$), under mild spectral-gap assumptions. It also provides an average-case bound under random perturbations (Theorem $t_i2$) and validates the qualitative model with numerical experiments comparing $\texttt{double}$, $\texttt{single}$, and $\texttt{half}$ precision, showing that small singular values can rise while large ones stay largely intact. The results offer a precision-induced regularization perspective with potential implications for numerical linear algebra and the design of mixed-precision computations.

Abstract

We perturb a real matrix $A$ of full column rank, and derive lower bounds for the smallest singular values of the perturbed matrix, in terms of normwise absolute perturbations. Our bounds, which extend existing lower-order expressions, demonstrate the potential increase in the smallest singular values, and represent a qualitative model for the increase in the small singular values after a matrix has been downcast to a lower arithmetic precision. Numerical experiments confirm the qualitative validity of this model and its ability to predict singular values changes in the presence of decreased arithmetic precision.

Small singular values can increase in lower precision

TL;DR

The paper addresses how downcasting a tall, full-column-rank matrix to lower arithmetic precision can increase its smallest singular values. It models precision demotion as a normwise perturbation and develops deterministic lower bounds for the smallest singular value, first for a single value via Theorems / and then for a cluster via Theorems / (and the related ), under mild spectral-gap assumptions. It also provides an average-case bound under random perturbations (Theorem ) and validates the qualitative model with numerical experiments comparing , , and precision, showing that small singular values can rise while large ones stay largely intact. The results offer a precision-induced regularization perspective with potential implications for numerical linear algebra and the design of mixed-precision computations.

Abstract

We perturb a real matrix of full column rank, and derive lower bounds for the smallest singular values of the perturbed matrix, in terms of normwise absolute perturbations. Our bounds, which extend existing lower-order expressions, demonstrate the potential increase in the smallest singular values, and represent a qualitative model for the increase in the small singular values after a matrix has been downcast to a lower arithmetic precision. Numerical experiments confirm the qualitative validity of this model and its ability to predict singular values changes in the presence of decreased arithmetic precision.
Paper Structure (34 sections, 11 theorems, 116 equations, 6 figures, 2 tables, 2 algorithms)

This paper contains 34 sections, 11 theorems, 116 equations, 6 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Motivation
  3. Modelling demotion to lower precision in terms of perturbations
  4. Our contributions
  5. Existing work
  6. Overview
  7. A single smallest singular value
  8. Auxiliary eigenvalue results
  9. A lower bound for the smallest singular value
  10. 1. Confirm that $\boldsymbol{C}_{11}$ satisfies the assumptions of Theorem \ref{['t_dp5']}
  11. 2. Derive the lower bound for $\mathop{\mathrm{\tilde{\lambda}_{\text{min}}}}\nolimits$ from Theorem \ref{['t_dp5']}
  12. A cluster of small singular values
  13. Auxiliary eigenvalue results
  14. Proof of (\ref{['e_dp2b']})
  15. Proof of (\ref{['e_dp2a']})
  16. ...and 19 more sections

Key Result

Theorem 1

Let $\boldsymbol{A}\in{\mathbb{R}}^{m \times n}$ with $m\geq n$ have $\mathop{\mathrm{\mathrm{rank}}}\nolimits(\boldsymbol{A})\geq n-r$ for some $r\geq 1$. Let $\boldsymbol{A}=\boldsymbol{U}\mathbf{\Sigma}\boldsymbol{V}^T$ be a full singular value decomposition, where $\mathbf{\Sigma}\in{\mathbb{R}} where $\mathbf{\Sigma}_1,\boldsymbol{E}_{11}\in\mathbb{R}^{(n-r)\times (n-r)}$ with $\mathbf{\Sigma

Figures (6)

  • Figure 1: The matrix $\boldsymbol{A}\in\mathbb{R}^{4096\times 256}$ has 255 distinct singular values in $[10^{-4}, 10^2]$, and a single small singular value $10^{-7}$. All panels: Double precision singular values (squares). Left: Exact singular values (triangles). Right: Single precision singular values (stars).
  • Figure 2: The matrix $\boldsymbol{A}\in\mathbb{R}^{4096\times 256}$ has 255 distinct singular values in $[10^{-1}, 10^1]$, and a single small singular value $10^{-3}$. All panels: Double precision singular values (squares). Left: Exact singular values (triangles). Right: Half precision singular values (stars).
  • Figure 3: The matrix $\boldsymbol{A}\in\mathbb{R}^{4096\times 256}$ has 228 distinct singular values in $[10^{-1}, 10^5]$, and 28 distinct singular values in $[10^{-5}, 10^{-3}]$. All panels: Double precision singular values (squares). Left: Exact singular values (triangles). Right: Single precision singular values (stars).
  • Figure 4: The matrix $\boldsymbol{A}\in\mathbb{R}^{4096\times 256}$ has 228 distinct singular values in $[10^{0}, 10^2]$, and 28 distinct singular values in $[10^{-4}, 10^{-2}]$. All panels: Double precision singular values (squares). Left: Exact singular values (triangles). Right: Half precision singular values (stars).
  • Figure 5: The matrix $\boldsymbol{A}\in\mathbb{R}^{4096\times 256}$ has 228 distinct singular values in $[10^{-3}, 10^4]$, and 28 distinct singular values in $[10^{-7}, 10^{-4}]$. All panels: Double precision singular values (squares). Left: Exact singular values (triangles). Right: Single precision singular values (stars).
  • ...and 1 more figures

Theorems & Definitions (23)

  • Theorem 1
  • Lemma 1: Exact expression
  • proof
  • Lemma 2: Exact expression with stronger assumption
  • proof
  • Theorem 2: First lower bound
  • proof
  • Theorem 3: Second lower bound
  • proof
  • Theorem 4
  • ...and 13 more