Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Adaptive Mixed Precision and Dynamically Scaled Preconditioned Conjugate Gradient Algorithm

Yichen Guo, Eric de Sturler, Tim Warburton

TL;DR

This work tackles the performance bottlenecks of solving SPD linear systems with PCG by introducing AMP-PCG, an adaptive mixed-precision method augmented with dynamic scaling. By separately controlling the precision for matvec/residual and for preconditioned residual/search directions, and by scaling vectors to prevent underflow in FP16, AMP-PCG preserves convergence rates and final accuracy comparable to double precision PCG while reducing cost. The authors derive an attainable accuracy indicator and a precision-switching criterion, analyze convergence under inexact Krylov subspaces, and validate the approach with extensive numerical experiments, including a large GPU-based finite element solve that achieves about a 1.63x speedup. The results highlight strong robustness and practical speedups, while also discussing limitations related to residual oscillations and threshold sensitivity, pointing to directions for more robust switching rules and preconditioning strategies.

Abstract

We propose an adaptive mixed precision and dynamically scaled preconditioned conjugate gradient algorithm (AMP-PCG). It dynamically adjusts the precision for storing vectors and computing, exploiting low precision when appropriate, while maintaining a convergence rate and accuracy comparable to that of double precision PCG. Our mixed precision strategy consists of three main components: (1) The residual and matrix-vector product are initially computed in double precision, and the algorithm switches these to single precision based on the chosen convergence tolerance and an estimate of the residual gap. (2) Depending on the eigenvalue distribution, the preconditioned residual and search direction are either in half precision throughout the iterations or initially in double precision and then stepwise reduced to single and half precision. (3) A dynamically scaled residual is used at every iteration to mitigate underflow in half precision. We provide theoretical support for our estimates and we demonstrate the effectiveness of AMP-PCG through numerical experiments, highlighting both its robustness and the significant performance gains (1.63x speedup) achieved compared to double precision PCG on a GPU.

An Adaptive Mixed Precision and Dynamically Scaled Preconditioned Conjugate Gradient Algorithm

TL;DR

This work tackles the performance bottlenecks of solving SPD linear systems with PCG by introducing AMP-PCG, an adaptive mixed-precision method augmented with dynamic scaling. By separately controlling the precision for matvec/residual and for preconditioned residual/search directions, and by scaling vectors to prevent underflow in FP16, AMP-PCG preserves convergence rates and final accuracy comparable to double precision PCG while reducing cost. The authors derive an attainable accuracy indicator and a precision-switching criterion, analyze convergence under inexact Krylov subspaces, and validate the approach with extensive numerical experiments, including a large GPU-based finite element solve that achieves about a 1.63x speedup. The results highlight strong robustness and practical speedups, while also discussing limitations related to residual oscillations and threshold sensitivity, pointing to directions for more robust switching rules and preconditioning strategies.

Abstract

We propose an adaptive mixed precision and dynamically scaled preconditioned conjugate gradient algorithm (AMP-PCG). It dynamically adjusts the precision for storing vectors and computing, exploiting low precision when appropriate, while maintaining a convergence rate and accuracy comparable to that of double precision PCG. Our mixed precision strategy consists of three main components: (1) The residual and matrix-vector product are initially computed in double precision, and the algorithm switches these to single precision based on the chosen convergence tolerance and an estimate of the residual gap. (2) Depending on the eigenvalue distribution, the preconditioned residual and search direction are either in half precision throughout the iterations or initially in double precision and then stepwise reduced to single and half precision. (3) A dynamically scaled residual is used at every iteration to mitigate underflow in half precision. We provide theoretical support for our estimates and we demonstrate the effectiveness of AMP-PCG through numerical experiments, highlighting both its robustness and the significant performance gains (1.63x speedup) achieved compared to double precision PCG on a GPU.
Paper Structure (21 sections, 2 theorems, 44 equations, 11 figures, 2 tables, 3 algorithms)

This paper contains 21 sections, 2 theorems, 44 equations, 11 figures, 2 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Literature review
  3. AMP-PCG
  4. Motivating example
  5. Analysis
  6. (P1) Attainable accuracy
  7. (P2) Rate of convergence
  8. (P3) Dynamic Scaling
  9. Adaptive mixed precision and dynamically scaled PCG algorithm
  10. Criterion for switching precision for ${\bf r}_k$ and ${\bf q}_k$
  11. Numerical Results
  12. Motivating example
  13. Impact of the precision of ${\bf r}_k$ and ${\bf q}_k$ on attainable accuracy
  14. Cases with linear convergence
  15. Matrices without large, outlying eigenvalues
  16. ...and 6 more sections

Key Result

Theorem 3.1

Let $\delta \geq \left\|{\bf e}_{k}\right\|_{{\bf M}^{-1}}/\left\|{\bf r}_{k}\right\|_{{\bf M}^{-1}}.$ If $\delta'= 2 \sin \theta \sqrt{\kappa} < 1$ with $\theta = \arcsin \delta$, $\kappa = \mathrm{cond}({\bf M}^{-1/2} {\bf A} {\bf M}^{-1/2})$, then IPCG converges, and for even $k$, where

Figures (11)

  • Figure 1: Convergence of (a) the updated residual, and (b) the true residual, computed as ${\bf b} - {\bf A}{\bf x}_k$, for PCG implemented in half, single, and double precision for \ref{['ex: motivating']}. The matrix ${\bf A}$ is defined in \ref{['eq:motivating A']}. Compared with FP64, both FP16 and FP32 yield lower attainable accuracy and slower convergence. In the FP16 case, $\rho_k$ eventually underflows to zero due to limited dynamic range, leading to breakdown, indicated by the red star. This example highlights that naive use of low precision in PCG can lead to reduced accuracy, delayed convergence, and data underflow.
  • Figure 1: Revisiting the motivating example (see \ref{['ex: motivating', 'fig: motivation example']}): convergence of (a) the relative updated residual norm and (b) the relative true residual norm for PCG(FP64) and AMP-PCG(FP64), as discussed in \ref{['sec: revisit motivating']}. Cross markers indicate the iterations where the precision $u_{z,k}$, used for $\mathbf{z}_k$ and $\mathbf{p}_k$, is reduced to single and then to half precision. The blue circle marks where $u_{r,k}$, used for $\mathbf{r}_k$ and $\mathbf{q}_k$, is reduced from double to single precision.
  • Figure 2: Convergence curves of the relative updated residual norm ((a), (c), and (e)) and the relative true residual norm ((b), (d), and (f)) for PCG(FP64) and AMP-PCG($u_{r,k}$) applied to \ref{['ex: motivating', 'ex:bcsstk', 'ex:622bus']}, respectively. Two stopping tolerances, $10^{-6}$ and $10^{-8}$, are tested. At the switch point, the precision of ${\bf r}_k$ and ${\bf q}_k$ is reduced from FP64 to FP32; all other vectors remain in FP64. The dotted curves show the indicator $\widehat{\eta}_k$ (see \ref{['eq: Weta_k']}), which provides an estimate of the final attainable accuracy.
  • Figure 3: Convergence of the relative residual norm for PCG(FP64) and AMP-PCG(FP16) on (a) \ref{['ex: linear conv1']} and (b) \ref{['ex: linear conv2']}. In AMP-PCG(FP16), ${\bf z}_k$ and ${\bf p}_k$ are in half precision throughout the iterations. The blue circle marks where $u_{r,k}$, used for $\mathbf{r}_k$ and $\mathbf{q}_k$, is reduced from double to single precision.
  • Figure 4: Convergence curves of the relative residual norm for PCG(FP64), AMP-PCG(FP64), and AMP-PCG(FP16) applied to \ref{['ex: not large outlying eigs', 'ex:bcsstk', 'ex:622bus']} in \ref{['sec: matcol']}. The red and black crosses indicate the iterations at which $u_{z,k}$, used for ${\bf z}_k$ and ${\bf p}_k$, is set to FP32 and FP16, respectively. The blue circle marks the iterations where $u_{r,k}$ is switched to FP32.
  • ...and 6 more figures

Theorems & Definitions (14)

  • Example 1
  • Theorem 3.1: Theorem 3.6 in golub1999inexact
  • Theorem 3.2
  • Remark 4.4
  • Example 2
  • Example 3
  • Example 4
  • Example 5
  • Example 6
  • Example 7
  • ...and 4 more