ZOQO: Zero-Order Quantized Optimization
Noga Bar, Raja Giryes
TL;DR
This work addresses the challenge of training under resource constraints by proposing ZOQO, a fully quantized zero-order optimization framework that eliminates the need for full-precision gradients. Building on ZO-SignSGD, ZOQO substitutes Gaussian perturbations with discretized quantized noise sampled from a bounded set $\mathcal{B}$ and estimates the gradient sign using forward queries via $\mathrm{sign}(\ell(\mathbf{x}^+) - \ell(\mathbf{x}^-))$ with $q=1$. A key contribution is the learning-rate scaling $\eta_q = c(\eta) = \max\{\lfloor \eta/s \rfloor, 1\} s$ to ensure updates occur in the quantized range, enabling all updates to be performed with quantized arithmetic: $\mathbf{x}_{t+1} - \mathbf{x}_t = -\eta_q\, \mathrm{sign}(\hat{\nabla} f)$. Empirical results on black-box adversarial attacks and LoRA-based fine-tuning of large language models demonstrate memory efficiency and competitive performance relative to full-precision baselines, highlighting ZOQO’s potential for edge devices and low-resource settings.
Abstract
The increasing computational and memory demands in deep learning present significant challenges, especially in resource-constrained environments. We introduce a zero-order quantized optimization (ZOQO) method designed for training models with quantized parameters and operations. Our approach leverages zero-order approximations of the gradient sign and adapts the learning process to maintain the parameters' quantization without the need for full-precision gradient calculations. We demonstrate the effectiveness of ZOQO through experiments in fine-tuning of large language models and black-box adversarial attacks. Despite the limitations of zero-order and quantized operations training, our method achieves competitive performance compared to full-precision methods, highlighting its potential for low-resource environments.