Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Quantization Robustness of Monotone Operator Equilibrium Networks

James Li, Philip H. W. Leong, Thomas Chaffey

TL;DR

This work analyzes weight quantization as a spectral perturbation of the underlying monotone inclusion, finding a phase transition at the predicted threshold: three- and four-bit post-training quantization diverge, while five-bit and above converge.

Abstract

Monotone operator equilibrium networks are implicit-layer models whose output is the unique equilibrium of a monotone operator, guaranteeing existence, uniqueness, and convergence. When deployed on low-precision hardware, weights are quantized, potentially destroying these guarantees. We analyze weight quantization as a spectral perturbation of the underlying monotone inclusion. Convergence of the quantized solver is guaranteed whenever the spectral-norm weight perturbation is smaller than the monotonicity margin; the displacement between quantized and full-precision equilibria is bounded in terms of the perturbation size and margin; and a condition number characterizing the ratio of the operator norm to the margin links quantization precision to forward error. MNIST experiments confirm a phase transition at the predicted threshold: three- and four-bit post-training quantization diverge, while five-bit and above converge. The backward-pass guarantee enables quantization-aware training, which recovers provable convergence at four bits.

Quantization Robustness of Monotone Operator Equilibrium Networks

TL;DR

This work analyzes weight quantization as a spectral perturbation of the underlying monotone inclusion, finding a phase transition at the predicted threshold: three- and four-bit post-training quantization diverge, while five-bit and above converge.

Abstract

Monotone operator equilibrium networks are implicit-layer models whose output is the unique equilibrium of a monotone operator, guaranteeing existence, uniqueness, and convergence. When deployed on low-precision hardware, weights are quantized, potentially destroying these guarantees. We analyze weight quantization as a spectral perturbation of the underlying monotone inclusion. Convergence of the quantized solver is guaranteed whenever the spectral-norm weight perturbation is smaller than the monotonicity margin; the displacement between quantized and full-precision equilibria is bounded in terms of the perturbation size and margin; and a condition number characterizing the ratio of the operator norm to the margin links quantization precision to forward error. MNIST experiments confirm a phase transition at the predicted threshold: three- and four-bit post-training quantization diverge, while five-bit and above converge. The backward-pass guarantee enables quantization-aware training, which recovers provable convergence at four bits.
Paper Structure (11 sections, 9 theorems, 14 equations, 3 figures)

This paper contains 11 sections, 9 theorems, 14 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Work
  4. Preliminaries
  5. Monotone operator equilibrium networks
  6. Quantization in a MonDEQ
  7. Margin Perturbation and Well-Posedness
  8. Equilibrium Displacement
  9. Backward Pass Under Quantization
  10. Numerical Experiments
  11. Conclusions

Key Result

Theorem 1

Define a MonDEQ as in Definition def:mondeq. Then $z^\star \in \operatorname{Fix}(\Phi) \Longleftrightarrow 0 \in F(z^\star) + G(z^\star)$.

Figures (3)

  • Figure 1: Margin stability certificate. Iterations to convergence (top) and final residual (bottom) vs. normalized perturbation $\left\lVert \Delta W \right\rVert_2/m$; each point is one bit-width (3--32 bits). The dashed line marks the sufficient condition $\left\lVert \Delta W \right\rVert_2/m = 1$. Circles: converged (relative residual ${<}\,10^{-5}$); crosses: did not converge within 2000 iterations.
  • Figure 2: Displacement bound validation (Theorem \ref{['thm:equilibrium_displacement']}) at 6, 8, 12, and 16 bits. Each point is one test sample ($x$-axis: theoretical bound $(\left\lVert \Delta W \right\rVert_2/m)\left\lVert \widetilde{z}^\star \right\rVert_2$; $y$-axis: empirical displacement $\left\lVert \widetilde{z}^\star - z^\star \right\rVert_2$). Points below the dashed line ($y = x$) satisfy the bound.
  • Figure 3: QAT vs. PTQ at 4, 6, and 8 bits. Left: test accuracy (%; a red X indicates PTQ non-convergence at 4 bits). Right: $\left\lVert \Delta W \right\rVert_2/m$; the dashed line marks $\left\lVert \Delta W \right\rVert_2/m = 1$.

Theorems & Definitions (19)

  • Definition 1
  • Theorem 1
  • Proposition 1
  • proof
  • Definition 2
  • Theorem 2
  • proof
  • Corollary 1
  • proof
  • Theorem 3
  • ...and 9 more