Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Algorithms and data structures for numerical computations with automatic precision estimation

Igor V. Netay

TL;DR

How to estimate precision for some collection of functions most commonly used for array manipulations and training of neural networks is described and a fast estimation of precision is provided for highly optimized functions like matrix multiplication.

Abstract

We introduce data structures and algorithms to count numerical inaccuracies arising from usage of floating numbers described in IEEE 754. Here we describe how to estimate precision for some collection of functions most commonly used for array manipulations and training of neural networks. For highly optimized functions like matrix multiplication, we provide a fast estimation of precision and some hint how the estimation can be strengthened.

Algorithms and data structures for numerical computations with automatic precision estimation

TL;DR

How to estimate precision for some collection of functions most commonly used for array manipulations and training of neural networks is described and a fast estimation of precision is provided for highly optimized functions like matrix multiplication.

Abstract

We introduce data structures and algorithms to count numerical inaccuracies arising from usage of floating numbers described in IEEE 754. Here we describe how to estimate precision for some collection of functions most commonly used for array manipulations and training of neural networks. For highly optimized functions like matrix multiplication, we provide a fast estimation of precision and some hint how the estimation can be strengthened.
Paper Structure (19 sections, 4 theorems, 15 equations, 2 tables)

This paper contains 19 sections, 4 theorems, 15 equations, 2 tables.

Table of Contents

  1. Numerical preliminaries
  2. Errors
  3. Error propagation in function computation
  4. Other reasons of results divergence
  5. Related work
  6. Isolation level for precision estimation
  7. Data structures and basic precision estimations
  8. Precision estimation for particular functions
  9. Addition, subtraction and sum
  10. Multiplication, division and product
  11. Rounding
  12. Maxima and minima
  13. Differentiable functions
  14. Matrix multiplication and tropical semiring
  15. Gradients in neural networks
  16. ...and 4 more sections

Key Result

Theorem 5.1

Let $A$, $B$, and $C$ are matrices such that $A \cdot B = C$ of shapes ($m \times n$, $n \times k$ and $m \times k$). Suppose that $A$ (resp. $B$, $C$) has inaccuracies $\mathcal{A}$ (resp. $\mathcal{B}$ and $\mathcal{C}$). Then In particular, the following inequality holds: where power of $2$ and $\log_2$ are applied element-wise.

Theorems & Definitions (12)

  • Definition 1.1
  • Definition 1.2
  • Example 1.3
  • Example 1.4
  • Theorem 5.1
  • proof
  • Theorem 5.2
  • proof
  • Remark 5.3
  • Theorem 6.1
  • ...and 2 more