Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stochastic Rounding Implicitly Regularizes Tall-and-Thin Matrices

Gregory Dexter, Christos Boutsikas, Linkai Ma, Ilse C. F. Ipsen, Petros Drineas

TL;DR

This work investigates stochastic rounding (SR) for tall-and-thin matrices and shows that SR implicitly regularizes such matrices by making the smallest singular value of the rounded matrix stay bounded away from zero with high probability. The authors derive a general lower bound $\sigma_d(\widetilde{\mathbf{A}}) \ge \mathcal{R}\sqrt{n}(\sqrt{\nu}-\varepsilon_{n,d})$, where $\nu$ captures the stochasticity in rounding and $\mathcal{R}$ bounds entrywise errors; under mild dimension growth ($d=O((n/\log n)^{1/4})$), $\varepsilon_{n,d}\to0$, yielding $\sigma_d(\widetilde{\mathbf{A}}) \gtrsim \mathcal{R}\sqrt{n\nu}$. The proofs combine anti-concentration results from Random Matrix Theory with projection-based Weyl-type arguments, ensuring errors do not concentrate in low-dimensional subspaces. Extensive experiments with fixed-point and floating-point SR across rank-deficient and full-rank matrices confirm the theoretical predictions and show practical relevance, though the current bounds can be conservative for modest $n/d$; a small constant relaxation aligns theory with observed behavior. Overall, SR provides a principled mechanism for implicit regularization in high-dimensional linear settings, with potential implications for training and deploying low-precision neural networks and large-scale ML models.

Abstract

Motivated by the popularity of stochastic rounding in the context of machine learning and the training of large-scale deep neural network models, we consider stochastic nearness rounding of real matrices $\mathbf{A}$ with many more rows than columns. We provide novel theoretical evidence, supported by extensive experimental evaluation that, with high probability, the smallest singular value of a stochastically rounded matrix is well bounded away from zero -- regardless of how close $\mathbf{A}$ is to being rank deficient and even if $\mathbf{A}$ is rank-deficient. In other words, stochastic rounding \textit{implicitly regularizes} tall and skinny matrices $\mathbf{A}$ so that the rounded version has full column rank. Our proofs leverage powerful results in random matrix theory, and the idea that stochastic rounding errors do not concentrate in low-dimensional column spaces.

Stochastic Rounding Implicitly Regularizes Tall-and-Thin Matrices

TL;DR

This work investigates stochastic rounding (SR) for tall-and-thin matrices and shows that SR implicitly regularizes such matrices by making the smallest singular value of the rounded matrix stay bounded away from zero with high probability. The authors derive a general lower bound , where captures the stochasticity in rounding and bounds entrywise errors; under mild dimension growth (), , yielding . The proofs combine anti-concentration results from Random Matrix Theory with projection-based Weyl-type arguments, ensuring errors do not concentrate in low-dimensional subspaces. Extensive experiments with fixed-point and floating-point SR across rank-deficient and full-rank matrices confirm the theoretical predictions and show practical relevance, though the current bounds can be conservative for modest ; a small constant relaxation aligns theory with observed behavior. Overall, SR provides a principled mechanism for implicit regularization in high-dimensional linear settings, with potential implications for training and deploying low-precision neural networks and large-scale ML models.

Abstract

Motivated by the popularity of stochastic rounding in the context of machine learning and the training of large-scale deep neural network models, we consider stochastic nearness rounding of real matrices with many more rows than columns. We provide novel theoretical evidence, supported by extensive experimental evaluation that, with high probability, the smallest singular value of a stochastically rounded matrix is well bounded away from zero -- regardless of how close is to being rank deficient and even if is rank-deficient. In other words, stochastic rounding \textit{implicitly regularizes} tall and skinny matrices so that the rounded version has full column rank. Our proofs leverage powerful results in random matrix theory, and the idea that stochastic rounding errors do not concentrate in low-dimensional column spaces.
Paper Structure (30 sections, 8 theorems, 71 equations, 6 figures, 6 tables)

This paper contains 30 sections, 8 theorems, 71 equations, 6 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Our results
  3. Background
  4. Notation
  5. Stochastic rounding and its properties
  6. Normalized floating point numbers
  7. Bounding the sum of random variables
  8. The union bound
  9. Prior work
  10. Bounding the smallest singular value of the perturbed matrix
  11. A simple example
  12. General random perturbation
  13. Perturbations from SR-nearness
  14. A Random Matrix Theory bound
  15. Proof of Theorem \ref{['lemma:general_rounding']}
  16. ...and 15 more sections

Key Result

Theorem 1

Let $X_1,\ldots,X_n$ be independent random variables with $m_i\leq X_i \leq M_i$, $1\leq i\leq n$. Then, for any $t > 0$,

Figures (6)

  • Figure 1: The elements of $\mathbf{A}$ are random variables in $\mathcal{N}(0,1)$ with $\sigma_d(\mathbf{A})=0$. The stochastically rounded $\widetilde{\mathbf{A}}$ has elements in $\mathcal{F}^{{p}}$, for $p = 1, 2, 3$. The horizontal axis represents the values of $\sigma_d(\widetilde{\mathbf{A}})$ over 100 runs, grouped into at most 10 bins. The vertical axis represents the number of $\sigma_d(\widetilde{\mathbf{A}})$ in each bin. The orange dashed vertical line represents the average value of $\sigma_d(\widetilde{\mathbf{A}})$, while the red dashed vertical line represents the lower bound estimate (\ref{['eq:maineqapprox']}). Each panel corresponds to a different combination of $p$ and $d$. In each row, the precision $p$ is fixed, while the column dimension $d$ varies.
  • Figure 2: The matrices are initially drawn from a log-normal distribution with the smallest singular value set to 0, and stochastically rounded to $\mathcal{F}^{{p}}$, for $p = 1, \ldots, 3$. The horizontal axis represents the distribution of $\sigma_d(\widetilde{\mathbf{A}})$ over 100 repetitions, grouped into up to 10 bins. The vertical axis shows the frequency with which each $\sigma_d(\widetilde{\mathbf{A}})$ appears in each bin. The orange dashed vertical line represents the average value of $\sigma_d(\widetilde{\mathbf{A}})$, while the red dashed vertical line represents the lower bound estimate (\ref{['eq:maineqapprox']}). Each panel corresponds to a different combination of $p$ and $d$. In each row, the precision $p$ is fixed, while the column dimension $d$ varies.
  • Figure 3: The matrices are initially drawn from a standard normal distribution with the smallest singular value set to $10^{-2}$, and stochastically rounded to $\mathcal{F}^{{p}}$, for $p = 1, \ldots, 3$. The horizontal axis represents the distribution of $\sigma_d(\widetilde{\mathbf{A}})$ over 100 repetitions, grouped into up to 10 bins. The vertical axis shows the frequency with which each $\sigma_d(\widetilde{\mathbf{A}})$ appears in each bin. The orange dashed vertical line represents the average value of $\sigma_d(\widetilde{\mathbf{A}})$, while the red dashed vertical line represents the lower bound estimate (\ref{['eq:maineqapprox']}). Each panel corresponds to a different combination of $p$ and $d$. In each row, the precision $p$ is fixed, while the column dimension $d$ varies.
  • Figure 4: The matrices are initially drawn from a log-normal distribution with the smallest singular value set to $10^{-2}$, and stochastically rounded to $\mathcal{F}^{{p}}$, for $p = 1, \ldots, 3$. The horizontal axis represents the distribution of $\sigma_d(\widetilde{\mathbf{A}})$ over 100 repetitions, grouped into up to 10 bins. The vertical axis shows the frequency with which each $\sigma_d(\widetilde{\mathbf{A}})$ appears in each bin. The orange dashed vertical line represents the average value of $\sigma_d(\widetilde{\mathbf{A}})$, while the red dashed vertical line represents the lower bound estimate (\ref{['eq:maineqapprox']}). Each panel corresponds to a different combination of $p$ and $d$. In each row, the precision $p$ is fixed, while the column dimension $d$ varies.
  • Figure 5: The matrices are initially drawn from both a standard normal and a log-norm distribution with the smallest singular value set to 0, and stochastically rounded to single precision. The horizontal axis represents the distribution of $\sigma_d(\widetilde{\mathbf{A}})$ over 100 repetitions, grouped into up to 10 bins. The vertical axis shows the frequency with which each $\sigma_d(\widetilde{\mathbf{A}})$ appears in each bin. The orange dashed vertical line represents the average value of $\sigma_d(\widetilde{\mathbf{A}})$, while the red dashed vertical line represents the lower bound estimate (\ref{['eq:maineqapprox']}). Each panel corresponds to a different combination of the initial distribution and $d$. In each row, column dimension $d$ is fixed, while the distribution varies.
  • ...and 1 more figures

Theorems & Definitions (10)

  • Theorem 1: Theorem 2 in Hoeff63
  • Theorem 2
  • Corollary 3
  • Theorem 4: Theorem 2.10 and Remark 2.11 in dumitriu2022extreme
  • Example 1
  • Theorem 5
  • Remark 6
  • Theorem 7
  • Theorem 8
  • Lemma B.1