Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Irrational Complex Rotations Empower Low-bit Optimizers

Zhen Tian, Wayne Xin Zhao, Ji-Rong Wen

TL;DR

This work tackles the memory burden of optimizer states in training large models by introducing π-Quant, a memory-efficient optimizer that leverages irrational complex rotations to encode parameter pairs as a single rotation angle. The authors prove a data-compression theorem allowing two real values to be represented by one angle, then develop a linear-time geometric method to recover the angle and a quantization pipeline that reduces angle precision to 2λ digits, yielding about $3.32\times$bit-width per parameter. Empirically, π-Quant achieves full accuracy with as little as $3.32$-bit optimizer states and substantial memory reductions (e.g., ~41.9% on TinyLlama/GPU memory) across language modeling and diverse downstream tasks, often outperforming existing quantizers. The approach emphasizes non-uniform error distribution and compatibility with existing optimizers, offering a practical path to memory-efficient training for large models. Overall, π-Quant provides a theoretically grounded, scalable method for low-bit optimizer state representation with meaningful impact on training efficiency.

Abstract

In this paper, we propose a novel optimizer state compression algorithm, namely $π$-Quant, which leverages the properties of irrational numbers (e.g., $π$) for memory-efficient training. The core idea is based on our mathematical findings, which show that a pair of parameters can be represented by a single rotation angle using the complex rotation scheme. Building on this insight, we map the parameters into a complex space and perform quantization using the corresponding rotation angles. To efficiently integrate it into optimization process, we develop an efficient system of geometric equations that computes the precise rotation angles with linear complexity. We evaluate $π$-Quant on a wide range of tasks. Our experiments show that it can reduce the bit-width of parameters to 3.32-bit, achieving a 75% reduction in parameter scale and a 40% decrease in GPU memory usage, all while maintaining full accuracy.

Irrational Complex Rotations Empower Low-bit Optimizers

TL;DR

This work tackles the memory burden of optimizer states in training large models by introducing π-Quant, a memory-efficient optimizer that leverages irrational complex rotations to encode parameter pairs as a single rotation angle. The authors prove a data-compression theorem allowing two real values to be represented by one angle, then develop a linear-time geometric method to recover the angle and a quantization pipeline that reduces angle precision to 2λ digits, yielding about bit-width per parameter. Empirically, π-Quant achieves full accuracy with as little as -bit optimizer states and substantial memory reductions (e.g., ~41.9% on TinyLlama/GPU memory) across language modeling and diverse downstream tasks, often outperforming existing quantizers. The approach emphasizes non-uniform error distribution and compatibility with existing optimizers, offering a practical path to memory-efficient training for large models. Overall, π-Quant provides a theoretically grounded, scalable method for low-bit optimizer state representation with meaningful impact on training efficiency.

Abstract

In this paper, we propose a novel optimizer state compression algorithm, namely -Quant, which leverages the properties of irrational numbers (e.g., ) for memory-efficient training. The core idea is based on our mathematical findings, which show that a pair of parameters can be represented by a single rotation angle using the complex rotation scheme. Building on this insight, we map the parameters into a complex space and perform quantization using the corresponding rotation angles. To efficiently integrate it into optimization process, we develop an efficient system of geometric equations that computes the precise rotation angles with linear complexity. We evaluate -Quant on a wide range of tasks. Our experiments show that it can reduce the bit-width of parameters to 3.32-bit, achieving a 75% reduction in parameter scale and a 40% decrease in GPU memory usage, all while maintaining full accuracy.
Paper Structure (19 sections, 3 theorems, 30 equations, 7 figures, 4 tables, 2 algorithms)

This paper contains 19 sections, 3 theorems, 30 equations, 7 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Methodology
  4. Parameter Compression with Irrational Rotations
  5. Optimization with Rotation Quantization
  6. Discussion
  7. Experiment
  8. Language Modeling Evaluation Under Different Bit-Widths
  9. Experimental Analysis
  10. Accuracy on Other Downstream Tasks
  11. Ablation Study
  12. Visualizing the Gradient Update Process
  13. Related Work
  14. Conclusion
  15. Proof of Theorem \ref{['thm:irr']}.
  16. ...and 4 more sections

Key Result

Theorem 3.1

Given a complex number $z = x + iy \in \mathbb{C}$ that satisfies $||z|| \leq 2$, there exists a unique angle $\theta \in \mathbb{R}$ that rewrites $z$: where $i$ is the imaginary unit, and $\bar{\pi}$ denotes any irrational number (e.g., the circle ratio $\pi$).

Figures (7)

  • Figure 1: The overall framework of $\pi$-Quant.
  • Figure 2: The error distribution (top) and precision distribution (bottom) between our method and traditional methods.
  • Figure 3: Loss comparison between FP32 and our approach.
  • Figure 4: Ablation Study of the $\pi$ setting in our approach.
  • Figure 5: Visualizing the gradient descent process.
  • ...and 2 more figures

Theorems & Definitions (4)

  • Theorem 3.1
  • Lemma 3.2
  • Lemma 1.1
  • proof