Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Quantized Evolution Strategies: High-precision Fine-tuning of Quantized LLMs at Low-precision Cost

Yinggan Xu, Risto Miikkulainen, Xin Qiu

TL;DR

This work tackles the barrier to fine-tuning quantized LLMs by introducing Quantized Evolution Strategies (QES), a backpropagation-free framework that optimizes discrete weights directly in quantized space. QES combines accumulated error feedback, inspired by Delta-Sigma modulation, with a Stateless Seed Replay mechanism to preserve high-precision gradient signals while keeping memory usage at inference-like levels. Empirical results on a Countdown arithmetic reasoning task show that QES substantially surpasses zeroth-order baselines like QuZO across INT4, INT8, and W8A8 formats, achieving performance close to memory-intensive full-residual approaches. The approach promises scalable, hardware-friendly fine-tuning of large models by enabling full-parameter optimization within quantized spaces on commodity hardware.

Abstract

Post-Training Quantization (PTQ) is essential for deploying Large Language Models (LLMs) on memory-constrained devices, yet it renders models static and difficult to fine-tune. Standard fine-tuning paradigms, including Reinforcement Learning (RL), fundamentally rely on backpropagation and high-precision weights to compute gradients. Thus they cannot be used on quantized models, where the parameter space is discrete and non-differentiable. While Evolution Strategies (ES) offer a backpropagation-free alternative, optimization of the quantized parameters can still fail due to vanishing or inaccurate gradient. This paper introduces Quantized Evolution Strategies (QES), an optimization paradigm that performs full-parameter fine-tuning directly in the quantized space. QES is based on two innovations: (1) it integrates accumulated error feedback to preserve high-precision gradient signals, and (2) it utilizes a stateless seed replay to reduce memory usage to low-precision inference levels. QES significantly outperforms the state-of-the-art zeroth-order fine-tuning method on arithmetic reasoning tasks, making direct fine-tuning for quantized models possible. It therefore opens up the possibility for scaling up LLMs entirely in the quantized space. The source code is available at https://github.com/dibbla/Quantized-Evolution-Strategies .

Quantized Evolution Strategies: High-precision Fine-tuning of Quantized LLMs at Low-precision Cost

TL;DR

This work tackles the barrier to fine-tuning quantized LLMs by introducing Quantized Evolution Strategies (QES), a backpropagation-free framework that optimizes discrete weights directly in quantized space. QES combines accumulated error feedback, inspired by Delta-Sigma modulation, with a Stateless Seed Replay mechanism to preserve high-precision gradient signals while keeping memory usage at inference-like levels. Empirical results on a Countdown arithmetic reasoning task show that QES substantially surpasses zeroth-order baselines like QuZO across INT4, INT8, and W8A8 formats, achieving performance close to memory-intensive full-residual approaches. The approach promises scalable, hardware-friendly fine-tuning of large models by enabling full-parameter optimization within quantized spaces on commodity hardware.

Abstract

Post-Training Quantization (PTQ) is essential for deploying Large Language Models (LLMs) on memory-constrained devices, yet it renders models static and difficult to fine-tune. Standard fine-tuning paradigms, including Reinforcement Learning (RL), fundamentally rely on backpropagation and high-precision weights to compute gradients. Thus they cannot be used on quantized models, where the parameter space is discrete and non-differentiable. While Evolution Strategies (ES) offer a backpropagation-free alternative, optimization of the quantized parameters can still fail due to vanishing or inaccurate gradient. This paper introduces Quantized Evolution Strategies (QES), an optimization paradigm that performs full-parameter fine-tuning directly in the quantized space. QES is based on two innovations: (1) it integrates accumulated error feedback to preserve high-precision gradient signals, and (2) it utilizes a stateless seed replay to reduce memory usage to low-precision inference levels. QES significantly outperforms the state-of-the-art zeroth-order fine-tuning method on arithmetic reasoning tasks, making direct fine-tuning for quantized models possible. It therefore opens up the possibility for scaling up LLMs entirely in the quantized space. The source code is available at https://github.com/dibbla/Quantized-Evolution-Strategies .
Paper Structure (18 sections, 11 equations, 3 figures, 3 tables, 2 algorithms)

This paper contains 18 sections, 11 equations, 3 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM Quantization and Low-Bit Fine-Tuning
  4. Zeroth Order Fine-tuning for Quantized Models
  5. Techniques for Reducing Discretization Errors
  6. Method
  7. The Optimization Challenge
  8. Accumulated Error Feedback
  9. Stateless Error Tracking via Seed Replay
  10. Experiments
  11. Experimental Setup
  12. Performance
  13. Effect of Accumulation Hyperparameters
  14. Fidelity of Stateless Seed Replay
  15. Accelerating State Reconstruction
  16. ...and 3 more sections

Figures (3)

  • Figure 1: An overview of Quantized Evolution Strategies (QES). The goal is to optimize quantized LLMs directly on discrete parameter space in a memory-efficient manner. Error residuals are accumulated at each iteration until they reach a threshold for making a discrete change. QES achieves temporal equivalence to high-precision optimization trajectory, while maintaining a memory at the same level as inference-only quantized models.
  • Figure 2: Training curves for QUZO, QES, and Full-Residual QES compared to the Base Model. QuZO (Orange) performance is unstable and training collapses especially in the coarser INT4 landscape and with the smaller base model. In contrast, QES (Green) progresses steadily closely tracking the Full Residual Oracle (Blue) despite using significantly less memory.
  • Figure 3: A continuous reward function and its instantiation on a discrete grid. While the reward function can be optimized with a high-precision gradient ascent, that method cannot be applied to the discrete case. Instead, the gradient needs to be estimated without vanishing signals or inaccurate moves between grid points. That is the challenge that QES is designed to solve.