ECO: Quantized Training without Full-Precision Master Weights

TL;DR

The Error-Compensating Optimizer (ECO) is introduced, which eliminates master weights by applying updates directly to quantized parameters and carefully injects the resulting quantization error into the optimizer momentum, forming an error-feedback loop with no additional memory.

Abstract

Quantization has significantly improved the compute and memory efficiency of Large Language Model (LLM) training. However, existing approaches still rely on accumulating their updates in high-precision: concretely, gradient updates must be applied to a high-precision weight buffer, known as $\textit{master weights}$. This buffer introduces substantial memory overhead, particularly for Sparse Mixture of Experts (SMoE) models, where model parameters and optimizer states dominate memory usage. To address this, we introduce the Error-Compensating Optimizer (ECO), which eliminates master weights by applying updates directly to quantized parameters. ECO quantizes weights after each step and carefully injects the resulting quantization error into the optimizer momentum, forming an error-feedback loop with no additional memory. We prove that, under standard assumptions and a decaying learning rate, ECO converges to a constant-radius neighborhood of the optimum, while naive master-weight removal can incur an error that is inversely proportional to the learning rate. We show empirical results for pretraining small Transformers (30-800M), a Gemma-3 1B model, and a 2.1B parameter Sparse MoE model with FP8 quantization, and fine-tuning DeepSeek-MoE-16B in INT4 precision. Throughout, ECO matches baselines with master weights up to near-lossless accuracy, significantly shifting the static memory vs validation loss Pareto frontier.

Figures (3)

• Figure 1: Static Memory Used vs Validation Loss comparing the standard BF16, FP8 with Master weights (FP8 w/ MW) baselines with standard stochastic rounding (FP8 w/o MW + SR) and ECO. ECO with stochastic rounding (SR) provides a significantly better Pareto frontier. Gradient accumulation is disabled in all cases.
• Figure 2: Similarity of consecutive quantization errors. Left: relative norm $\left\lVert\mathbf{e}_{t+1}\right\rVert_2/\left\lVert\mathbf{e}_t\right\rVert_2$. Right: cosine similarity between $\mathbf{e}_t$ and $\mathbf{e}_{t+1}$.
• Figure 3: (Left) Gemma 3 1B and SMoE 2.1B validation loss comparison, and (Right) Smoothed training loss during fine-tuning of DeepSeek-MoE-16B-Base deepseekmoe.

Theorems & Definitions (15)

• Definition 3.4: Virtual Sequence
• Lemma 3.5: Virtual Sequence Dynamics
• Lemma 3.6: Descent Lemma
• Lemma 3.7: Bounded Momentum
• Theorem 3.8: Convergence Rate
• Lemma 3.9: Deterministic Momentum Bound
• Theorem 3.10: Deterministic Convergence
• proof
• proof
• proof