Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Recent and Upcoming Developments in Randomized Numerical Linear Algebra for Machine Learning

Michał Dereziński, Michael W. Mahoney

TL;DR

This survey surveys RandNLA from its classical foundations—sketching for matrix multiplication, LS, and LR—to modern theory that leverages Random Matrix Theory (RMT) and data-aware sampling. It outlines three optimization paradigms (Sketch-and-Solve, Iterative Sketching, Sketch-and-Precondition) and discusses advances in ML/Statistics, including robust learning, kernel methods via Nyström, bootstrap-based inference, and implicit regularization. The authors connect these algorithmic ideas to practical software, hardware trends, and next-generation libraries (RandBLAS/RandLAPACK), arguing that modern RandNLA provides tighter theory-practice alignment and broader applicability in ML, optimization, and data analysis. Overall, the paper emphasizes how randomness can be tightly integrated into both algorithms and systems to enable scalable, robust, and interpretable large-scale data analysis.

Abstract

Large matrices arise in many machine learning and data analysis applications, including as representations of datasets, graphs, model weights, and first and second-order derivatives. Randomized Numerical Linear Algebra (RandNLA) is an area which uses randomness to develop improved algorithms for ubiquitous matrix problems. The area has reached a certain level of maturity; but recent hardware trends, efforts to incorporate RandNLA algorithms into core numerical libraries, and advances in machine learning, statistics, and random matrix theory, have lead to new theoretical and practical challenges. This article provides a self-contained overview of RandNLA, in light of these developments.

Recent and Upcoming Developments in Randomized Numerical Linear Algebra for Machine Learning

TL;DR

This survey surveys RandNLA from its classical foundations—sketching for matrix multiplication, LS, and LR—to modern theory that leverages Random Matrix Theory (RMT) and data-aware sampling. It outlines three optimization paradigms (Sketch-and-Solve, Iterative Sketching, Sketch-and-Precondition) and discusses advances in ML/Statistics, including robust learning, kernel methods via Nyström, bootstrap-based inference, and implicit regularization. The authors connect these algorithmic ideas to practical software, hardware trends, and next-generation libraries (RandBLAS/RandLAPACK), arguing that modern RandNLA provides tighter theory-practice alignment and broader applicability in ML, optimization, and data analysis. Overall, the paper emphasizes how randomness can be tightly integrated into both algorithms and systems to enable scalable, robust, and interpretable large-scale data analysis.

Abstract

Large matrices arise in many machine learning and data analysis applications, including as representations of datasets, graphs, model weights, and first and second-order derivatives. Randomized Numerical Linear Algebra (RandNLA) is an area which uses randomness to develop improved algorithms for ubiquitous matrix problems. The area has reached a certain level of maturity; but recent hardware trends, efforts to incorporate RandNLA algorithms into core numerical libraries, and advances in machine learning, statistics, and random matrix theory, have lead to new theoretical and practical challenges. This article provides a self-contained overview of RandNLA, in light of these developments.
Paper Structure (24 sections, 47 equations, 3 figures)

This paper contains 24 sections, 47 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Foundations of "Classical" RandNLA
  3. Matrix Multiplication
  4. Least-squares Approximation
  5. Low-rank approximation
  6. Foundations of "Modern" RandNLA
  7. Algorithmic Gaussianization via RMT and LESS Embeddings
  8. RMT for Sampling via DPPs
  9. Advances in RandNLA for Optimization
  10. Gradient Sketch
  11. Hessian Sketch
  12. Sketch-and-Project
  13. Advances in RandNLA for ML and Statistics
  14. Statistical Learning Approaches
  15. Robust semi-supervised learning.
  16. ...and 9 more sections

Figures (3)

  • Figure 1: Schematic of three algorithmic paradigms for the use of RandNLA methods: "Classical" (subspace embedding based) RandNLA theory is most appropriate for low precision (with Sketch-and-Solve) and high precision (with Sketch-and-Precondition); while "Modern" (RMT-based) RandNLA theory is well-suited for moderate precision (with Iterative Sketching), which is of increasing interest in modern ML applications.
  • Figure 2: An illustration of the connection between RandNLA and statistical inference. Even though the input matrix $A$ for a RandNLA algorithm is often arbitrary and deterministic, the Algorithmic Gaussianization effect of sketching turns this matrix into a random data sample $\tilde{ A}$ (the sketch), which follows a generative model like those in statistical inference.
  • Figure 3: Fine-grained RMT analysis of LR approximation reveals a multiple-descent phenomenon that is not predicted by worst-case analysis.

Theorems & Definitions (2)

  • Definition 1
  • Definition 2