Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Forward and backward error bounds for a mixed precision preconditioned conjugate gradient algorithm

Thomas Bake, Erin Carson, Yuxin Ma

TL;DR

This work addresses finite-precision solution of SPD linear systems $Ax=b$ by developing a unified mixed-precision PCG framework that handles left, right, and split preconditioning. It proves that the relative backward error can be bounded by $O(u)$ and the forward error by $O(u)\kappa(A)^{1/2}$ after sufficiently many iterations, without assuming tiny residual norms, and show that low-precision preconditioning can preserve accuracy under mild conditions. The authors provide detailed per-step rounding-error analyses, local-orthogonality results, and a global backward/forward-error theorem, with specialization to Cholesky-based preconditioners in mixed precisions. Numerical experiments validate the theory, demonstrating near-machine-precision BE/FE levels across configurations and highlighting the practical benefits of split preconditioning for mixed-precision PCG. Overall, the results enable faster, energy-efficient PCG for large SPD systems while maintaining rigorous accuracy guarantees, guiding precision choices and stopping criteria in practice.

Abstract

The preconditioned conjugate gradient (PCG) algorithm is one of the most popular algorithms for solving large-scale linear systems Ax = b, where A is a symmetric positive definite matrix. Rather than computing residuals directly, it updates the residual vectors recursively. Current analyses of the conjugate gradient (CG) algorithm in finite precision typically assume that the norm of the recursively updated residual goes orders of magnitude below the machine precision, focusing mainly on bounding the residual gap thereafter. This work introduces a framework for the PCG algorithm and provides rigorous proofs that the relative backward and forward errors of the computed results of PCG can reach the levels O(u) and O(u)κ(A)^{1/2}, respectively, after a sufficient number of iterations without relying on an assumption concerning the norm of the recursively updated residual, where u represents the unit roundoff and κ(A) is the condition number of A. Our PCG framework further shows that applying preconditioners in low precision does not compromise the accuracy of the final results, provided that reasonable conditions are satisfied. Our theoretical results are illustrated through a set of numerical experiments.

Forward and backward error bounds for a mixed precision preconditioned conjugate gradient algorithm

TL;DR

This work addresses finite-precision solution of SPD linear systems by developing a unified mixed-precision PCG framework that handles left, right, and split preconditioning. It proves that the relative backward error can be bounded by and the forward error by after sufficiently many iterations, without assuming tiny residual norms, and show that low-precision preconditioning can preserve accuracy under mild conditions. The authors provide detailed per-step rounding-error analyses, local-orthogonality results, and a global backward/forward-error theorem, with specialization to Cholesky-based preconditioners in mixed precisions. Numerical experiments validate the theory, demonstrating near-machine-precision BE/FE levels across configurations and highlighting the practical benefits of split preconditioning for mixed-precision PCG. Overall, the results enable faster, energy-efficient PCG for large SPD systems while maintaining rigorous accuracy guarantees, guiding precision choices and stopping criteria in practice.

Abstract

The preconditioned conjugate gradient (PCG) algorithm is one of the most popular algorithms for solving large-scale linear systems Ax = b, where A is a symmetric positive definite matrix. Rather than computing residuals directly, it updates the residual vectors recursively. Current analyses of the conjugate gradient (CG) algorithm in finite precision typically assume that the norm of the recursively updated residual goes orders of magnitude below the machine precision, focusing mainly on bounding the residual gap thereafter. This work introduces a framework for the PCG algorithm and provides rigorous proofs that the relative backward and forward errors of the computed results of PCG can reach the levels O(u) and O(u)κ(A)^{1/2}, respectively, after a sufficient number of iterations without relying on an assumption concerning the norm of the recursively updated residual, where u represents the unit roundoff and κ(A) is the condition number of A. Our PCG framework further shows that applying preconditioners in low precision does not compromise the accuracy of the final results, provided that reasonable conditions are satisfied. Our theoretical results are illustrated through a set of numerical experiments.

Paper Structure

This paper contains 11 sections, 10 theorems, 134 equations, 4 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Notation, background and related work
  3. A framework for the preconditioned CG algorithm
  4. The numerical stability of PCG
  5. Rounding error analysis for each step
  6. Local orthogonality
  7. Backward and forward error analysis of PCG
  8. Finite precision PCG using Cholesky factors as the left, right, or split preconditioner
  9. Finite precision PCG with the left, right, or split preconditioning.
  10. Numerical Experiments
  11. Conclusions

Key Result

Theorem 1

Assume that $\hat{x}_{k}$ and $\hat{r}_{k}$ are generated by Algorithm alg:pcg satisfying eq:xk and eq:hatrk1. The difference between the true residual $\Tilde{r}_k = b - A\hat{x}_{k}$ and the recursively updated residual $\hat{r}_{k}$ satisfies

Figures (4)

  • Figure 1: Convergence curves for left PCG using the arithmetics listed in Table \ref{['tab:arithmetics']} for the application of the preconditioners. Left: Left-hand side of \ref{['eq:thm-final:normtilder']}. Right: Left-hand side of \ref{['eq:thm-final:normA-xk-x']}.
  • Figure 2: Comparison of convergence curves for split PCG variants using combinations of double and single precisions for the application of the preconditioners. Left: Split PCG from Algorithm \ref{['alg:pcg']}. Right: Saad's split PCG variant.
  • Figure 3: Achieved backward and forward errors by split PCG for (first row) and corresponding number of iterations required (second row).
  • Figure 4: Convergence curves for split PCG using arithmetics listed in Table \ref{['tab:arithmetics']} excluding fp16.

Theorems & Definitions (23)

  • Theorem 1
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • proof
  • Lemma 4
  • proof
  • Lemma 5
  • ...and 13 more