Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Balancing Inexactness in Mixed Precision Matrix Computations

Erin Claire Carson

TL;DR

A few recent examples are presented which demonstrate the potential for the use of mixed precision in numerical linear algebra and matrix computations, potentially improving performance without a noticeable decrease in accuracy.

Abstract

Support for arithmetic in multiple precisions and number formats is becoming increasingly common in emerging high-performance architectures. From a computational scientist's perspective, our goal is to determine how and where we can safely exploit mixed precision computation in our codes to improve performance. One case where the use of low precision is natural, common in computational science, is when there are already other significant sources of ``inexactness'' present, e.g., discretization error, measurement error, or algorithmic approximation error. In such instances, analyzing the interaction of these different sources of inexactness can give insight into how the precisions of various computations should be chosen in order to ``balance'' the errors, potentially improving performance without a noticeable decrease in accuracy. We present a few recent examples of this approach which demonstrate the potential for the use of mixed precision in numerical linear algebra and matrix computations.

Balancing Inexactness in Mixed Precision Matrix Computations

TL;DR

A few recent examples are presented which demonstrate the potential for the use of mixed precision in numerical linear algebra and matrix computations, potentially improving performance without a noticeable decrease in accuracy.

Abstract

Support for arithmetic in multiple precisions and number formats is becoming increasingly common in emerging high-performance architectures. From a computational scientist's perspective, our goal is to determine how and where we can safely exploit mixed precision computation in our codes to improve performance. One case where the use of low precision is natural, common in computational science, is when there are already other significant sources of ``inexactness'' present, e.g., discretization error, measurement error, or algorithmic approximation error. In such instances, analyzing the interaction of these different sources of inexactness can give insight into how the precisions of various computations should be chosen in order to ``balance'' the errors, potentially improving performance without a noticeable decrease in accuracy. We present a few recent examples of this approach which demonstrate the potential for the use of mixed precision in numerical linear algebra and matrix computations.
Paper Structure (14 sections, 8 equations, 4 figures, 1 table, 3 algorithms)

This paper contains 14 sections, 8 equations, 4 figures, 1 table, 3 algorithms.

Table of Contents

  1. Abstract.
  2. Introduction
  3. Floating Point Numbers and Mixed Precision Hardware
  4. The Challenges of Low Precision
  5. Reduced Numerical Range.
  6. Invalid Error Bounds.
  7. Reduced Precision.
  8. A (Very) Brief History of Mixed Precision Iterative Refinement
  9. Balancing Sources of Inexactness
  10. Example 1: Mixed Precision Sparse Approximate Inverse Preconditioners
  11. Example 2: Mixed Precision Randomized Nyström Approximation
  12. Example 3: Mixed Precision Hierarchical Matrix Approximations
  13. Other Examples and Outlook
  14. Acknowledgements.

Figures (4)

  • Figure 1: Comparison of GMRES-IR with LU preconditioning (left) and SPAI preconditioning (right) for the matrix steam1 from SuiteSparse davis2011university. Numbers above markers give the number of GMRES iterations performed in each refinement step; in all cases the GMRES convergence tolerance was set to $10^{-6}$. For the top left plot, AMD ordering was applied before computing the LU factors. In all cases, we use GMRES-IR with $u_f=$ single, $u =$ double, $u_r=$ double.
  • Figure 2: Experiments for matrix bcsstm07. The left plot shows the scaled spectrum $\lambda_k/\lambda_1$ (black dotted), and the values of $\sqrt{n}u_p$ for half precision (cyan) and single precision (red). The heuristic indicates that the finite precision error should not dominate for $k$ less than the point where the black dotted line intersects the colored lines. The right plot shows the Frobenius norm of the mean total error, $\Vert A-\widehat{A}_N\Vert_F$, taken over five runs, for exact arithmetic (blue diamonds), $u_p,u=$ double (green circles), $u_p=$ single and $u=$ double (red $+$s), and $u_p=$ half and $u=$ double (cyan $\times$s).
  • Figure 3: A depiction of a HODLR matrix with $\ell=3$ levels
  • Figure 4: Storage savings of adaptive-precision HODLR matrices relative to uniform (double) precision HODLR matrices. The depth $\ell=8$; red bars correspond to $\varepsilon=10^{-7}$, green bars indicate $\varepsilon=10^{-4}$, and blue bars indicate $\varepsilon=10^{-1}$.