Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Perturbation Analysis for Preconditioned Normal Equations in Mixed Precision

James E. Garrison, Ilse C. F. Ipsen

Abstract

For real matrices of full column-rank, we analyze the conditioning of several types of normal equations that are preconditioned by a randomized preconditioner computed in lower precision. These include symmetrically preconditioned normal equations, half-preconditioned normal equations, seminormal equations and not-normal equations. Our perturbation bounds are realistic and informative, and suggest that the conditioning depends only mildly on the quality of the preconditioner; however, it does depend on the size of the least squares residual -- even if the normal equations do not originate from a least squares problem. We illustrate that a randomized preconditioner can deliver a solution accuracy comparable to that of Matlab's mldivide command, is efficient in practice, and well-suited to GPU implementations. For the computation of the preconditioner, we propose an automatic selection of the precision, based on a fast condition number estimation in lower precision.

Perturbation Analysis for Preconditioned Normal Equations in Mixed Precision

Abstract

For real matrices of full column-rank, we analyze the conditioning of several types of normal equations that are preconditioned by a randomized preconditioner computed in lower precision. These include symmetrically preconditioned normal equations, half-preconditioned normal equations, seminormal equations and not-normal equations. Our perturbation bounds are realistic and informative, and suggest that the conditioning depends only mildly on the quality of the preconditioner; however, it does depend on the size of the least squares residual -- even if the normal equations do not originate from a least squares problem. We illustrate that a randomized preconditioner can deliver a solution accuracy comparable to that of Matlab's mldivide command, is efficient in practice, and well-suited to GPU implementations. For the computation of the preconditioner, we propose an automatic selection of the precision, based on a fast condition number estimation in lower precision.
Paper Structure (24 sections, 10 theorems, 69 equations, 7 figures, 2 algorithms)

This paper contains 24 sections, 10 theorems, 69 equations, 7 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preconditioned Normal Equations (PNE).
  3. Half Preconditioned Normal Equations (HPNE).
  4. Contributions and Overview
  5. Overview.
  6. Notation
  7. Existing Work
  8. Preconditioned Normal Equations (PNE)
  9. Extension of an existing PNE perturbation bound to mixed precision
  10. Improved PNE perturbation bound
  11. Half-Preconditioned Normal Equations (HPNE)
  12. Extension an existing HPNE perturbation bound to mixed precision
  13. Improved HPNE perturbation bound
  14. Seminormal and not-normal equations
  15. Seminormal equations
  16. ...and 9 more sections

Key Result

Lemma 1

\newlabell_ls0 Let $\boldsymbol{A}, \boldsymbol{A}+\boldsymbol{E}\in{\mathbb{R}}^{m \times n}$ with $\mathop{\mathrm{\mathrm{rank}}}\nolimits(\boldsymbol{A})=\mathop{\mathrm{\mathrm{rank}}}\nolimits(\boldsymbol{A}+\boldsymbol{E})=n$ and $\epsilon_A\equiv \|\boldsymbol{E}\|/\|\boldsymbol{A}\|$. Let

Figures (7)

  • Figure 1: Relative errors $\|\boldsymbol{\hat{\boldsymbol{x}}}-\boldsymbol{x}_*\|/\|\boldsymbol{\hat{\boldsymbol{x}}}\|$ in the computed solutions $\boldsymbol{\hat{\boldsymbol{x}}}$ and perturbation bounds versus logarithm of relative least squares residuals $\|\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_*\|/(\|\boldsymbol{A}\|\|\boldsymbol{x}_*\|)$ for $\boldsymbol{A}\in\mathbb{R}^{1,000\times 100}$ with condition number $\kappa(\boldsymbol{A})=10^8$. Shown are the Matlab backslash solutions (red plusses); the HPNE solutions with a single precision preconditioner (blue squares); the generalized version \ref{['eq42']} of the perturbation bound ipsen2025solutionsquaresproblemsrandomized (green circles); and our new bound \ref{['eq32']} (black x's).
  • Figure 1: Relative errors in the computed solutions $\boldsymbol{\hat{\boldsymbol{x}}}$ and perturbation bounds versus logarithm of relative least squares residuals $\|\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_*\|/(\|\boldsymbol{A}\|\|\boldsymbol{x}_*\|)$ for matrices $\boldsymbol{A}\in\mathbb{R}^{6,000\times 100}$ with condition number $\kappa(\boldsymbol{A})=10^4$. Shown are the errors in the Matlab backslash solutions (red plusses); the solutions from the PNE (blue squares); the old bound \ref{['eq31']} (green circles) and the new bound \ref{['eq41']} (black x) with precision $u_1=u_2=2^{-52}$.
  • Figure 2: Relative errors in the computed solutions $\boldsymbol{\hat{\boldsymbol{x}}}$ and perturbation bounds versus logarithm of relative least squares residuals $\|\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_*\|/(\|\boldsymbol{A}\|\|\boldsymbol{x}_*\|)$ for matrices $\boldsymbol{A}\in\mathbb{R}^{6,000\times 100}$ with condition number $\kappa(\boldsymbol{A})=10^4$. Shown are the errors in the Matlab backslash solutions (red plusses); the solutions from the HPNE (blue squares); the old bound \ref{['eq42']} (green circles) and the new bound \ref{['eq32']} (black x) with precision $u_1=u_2=2^{-52}$.
  • Figure 3: Relative errors in the computed solutions $\boldsymbol{\hat{\boldsymbol{x}}}$ and perturbation bounds versus logarithm of relative least squares residuals $\|\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_*\|/(\|\boldsymbol{A}\|\|\boldsymbol{x}_*\|)$ for matrices with condition number $\kappa(\boldsymbol{A})=10^8$. Shown are the errors in the Matlab backslash solutions (red plusses); the solutions from the PNE with a single precision preconditioner (blue squares); the old bound \ref{['eq31']} (green circles) and the new bound \ref{['eq41']} (black x) with precisions $u_1=2^{-23}$ and $u_2=2^{-52}$.
  • Figure 4: Relative errors in the computed solutions $\boldsymbol{\hat{\boldsymbol{x}}}$ and perturbation bounds versus logarithm of relative least squares residuals $\|\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_*\|/(\|\boldsymbol{A}\|\|\boldsymbol{x}_*\|)$ for matrices with condition number $\kappa(\boldsymbol{A})=10^8$. Shown are the errors in the Matlab backslash solutions (red plusses); the solutions from the HPNE with a single precision preconditioner (blue squares); the old bound \ref{['eq42']} (green circles) and the new bound \ref{['eq32']} (black x) with precisions $u_1=u^{-23}$ and $u_2=u^{-52}$.
  • ...and 2 more figures

Theorems & Definitions (19)

  • Lemma 1: Fact 5.14 in IIbook
  • Lemma 2: Lemma A.1 in ipsen2025solutionsquaresproblemsrandomized
  • Theorem 1
  • Proof 1
  • Theorem 2
  • Proof 2
  • Theorem 1
  • Proof 3
  • Theorem 2
  • Proof 4
  • ...and 9 more