Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

One-Bit Quantization for Random Features Models

Danil Akhtiamov, Reza Ghane, Babak Hassibi

TL;DR

This work analyzes one-bit weight quantization in the Random Features framework, a tractable proxy for wide neural networks, to understand its impact on generalization. By combining Gaussian Universality and Gaussian Equivalence with the Convex Gaussian Min-Max Theorem, the authors derive asymptotically exact expressions for the test error across deep RFs trained by Stochastic Mirror Descent. They prove that quantizing all hidden layers to one bit does not degrade generalization in the proportional regime, yielding a deterministic limit $\tau^{(L)}$ identical to the full-precision case, while the last layer remains full-precision. Empirically, they observe near-identical performance between Gaussian and one-bit RFs on synthetic data and MNIST, and report practical $\sim$4x inference speedups on a laptop GPU, underscoring the practical value of extreme quantization for edge deployment. These results provide theoretical justification for aggressive weight compression in deep RFs and offer a foundation for extending compression-aware design to learnable representations and classification tasks.

Abstract

Recent advances in neural networks have led to significant computational and memory demands, spurring interest in one-bit weight compression to enable efficient inference on resource-constrained devices. However, the theoretical underpinnings of such compression remain poorly understood. We address this gap by analyzing one-bit quantization in the Random Features model, a simplified framework that corresponds to neural networks with random representations. We prove that, asymptotically, quantizing weights of all layers except the last incurs no loss in generalization error, compared to the full precision random features model. Our findings offer theoretical insights into neural network compression. We also demonstrate empirically that one-bit quantization leads to significant inference speed ups for the Random Features models even on a laptop GPU, confirming the practical benefits of our work. Additionally, we provide an asymptotically precise characterization of the generalization error for Random Features with an arbitrary number of layers. To the best of our knowledge, our analysis yields more general results than all previous works in the related literature.

One-Bit Quantization for Random Features Models

TL;DR

This work analyzes one-bit weight quantization in the Random Features framework, a tractable proxy for wide neural networks, to understand its impact on generalization. By combining Gaussian Universality and Gaussian Equivalence with the Convex Gaussian Min-Max Theorem, the authors derive asymptotically exact expressions for the test error across deep RFs trained by Stochastic Mirror Descent. They prove that quantizing all hidden layers to one bit does not degrade generalization in the proportional regime, yielding a deterministic limit identical to the full-precision case, while the last layer remains full-precision. Empirically, they observe near-identical performance between Gaussian and one-bit RFs on synthetic data and MNIST, and report practical 4x inference speedups on a laptop GPU, underscoring the practical value of extreme quantization for edge deployment. These results provide theoretical justification for aggressive weight compression in deep RFs and offer a foundation for extending compression-aware design to learnable representations and classification tasks.

Abstract

Recent advances in neural networks have led to significant computational and memory demands, spurring interest in one-bit weight compression to enable efficient inference on resource-constrained devices. However, the theoretical underpinnings of such compression remain poorly understood. We address this gap by analyzing one-bit quantization in the Random Features model, a simplified framework that corresponds to neural networks with random representations. We prove that, asymptotically, quantizing weights of all layers except the last incurs no loss in generalization error, compared to the full precision random features model. Our findings offer theoretical insights into neural network compression. We also demonstrate empirically that one-bit quantization leads to significant inference speed ups for the Random Features models even on a laptop GPU, confirming the practical benefits of our work. Additionally, we provide an asymptotically precise characterization of the generalization error for Random Features with an arbitrary number of layers. To the best of our knowledge, our analysis yields more general results than all previous works in the related literature.
Paper Structure (25 sections, 6 theorems, 145 equations, 5 figures)

This paper contains 25 sections, 6 theorems, 145 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Lipschitz Concentration Property
  4. Problem Setting
  5. Stochastic Mirror Descent and Implicit Regularization
  6. Gaussian Universality
  7. Gaussian Equivalence
  8. Related Works
  9. Main Results
  10. Our approach and contributions
  11. Numerical Experiments
  12. One-Bit Quantization
  13. Synthetic data
  14. MNIST
  15. Inference speedup
  16. ...and 10 more sections

Key Result

Theorem 1

Let $\mathbf{X} \in \mathbb{R}^{n \times d}$ be a feature matrix whose rows are sampled from a disribution $\mathbb{P}$ with mean $\bm{\mu} \in \mathbb{R}^d$ and covariance $\bm{\Sigma} \in \mathbb{R}^{d\times d}$ and $\mathbf{y} \in \mathbb{R}^n$ be the labels satisfying Assumptions ass: univ. Let Then, asymptotically, the following holds for any Lipschitz function $g$ in probability: In partic

Figures (5)

  • Figure 1: Random Features with varying depth for Synthetic Data for SGD
  • Figure 2: Random Features with varying depth for Synthetic Data for Negative Entropy Mirror
  • Figure 3: Random Features with varying depth for MNIST
  • Figure 4: Random Features with varying hidden width for MNIST
  • Figure 5: Inference speed up vs Hidden Dimension

Theorems & Definitions (17)

  • Definition 1: Lipschitz Concentration Property
  • Theorem 1: ghane2024universality
  • Theorem 2
  • Remark 1
  • Remark 2
  • Remark 3
  • Definition 2
  • Definition 3: Lipschitz Concentration Property
  • Remark 4
  • Lemma 1
  • ...and 7 more