Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuning

Shuo Wang, Ziyu Chen, Ming Tang

Abstract

Fine-tuning large language models (LLMs) with backpropagation achieves high performance but incurs substantial memory overhead, limiting scalability on resource-constrained hardware. Zeroth-order (ZO) optimization provides a memory-efficient alternative by relying solely on forward passes, yet it typically suffers from slow or unstable convergence due to high-variance gradient estimates. Sparse ZO updates partially address this issue by perturbing only a subset of parameters, but their effectiveness hinges on selecting informative parameters, which is challenging in ZO optimization because each query yields only scalar feedback. We propose \textbf{Adaptive Curvature-Guided Sparse Zeroth-Order Optimization (CurvZO)}, which tracks curvature signals online from scalar ZO feedback and leverages these signals to construct a parameter-wise sampling distribution for selecting coordinates at each update, reducing the variance of the sparse ZO gradient estimator. Moreover, CurvZO dynamically adapts the perturbation budget to the evolving curvature signal distribution, yielding sparse ZO updates that remain both focused and sufficiently exploratory. Extensive experiments on OPT and Llama across diverse NLP tasks show that CurvZO consistently improves fine-tuning performance and reduces training time over ZO baselines. It improves accuracy by up to 4.4 points and achieves up to a $2\times$ speedup, while preserving memory efficiency.

CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuning

Abstract

Fine-tuning large language models (LLMs) with backpropagation achieves high performance but incurs substantial memory overhead, limiting scalability on resource-constrained hardware. Zeroth-order (ZO) optimization provides a memory-efficient alternative by relying solely on forward passes, yet it typically suffers from slow or unstable convergence due to high-variance gradient estimates. Sparse ZO updates partially address this issue by perturbing only a subset of parameters, but their effectiveness hinges on selecting informative parameters, which is challenging in ZO optimization because each query yields only scalar feedback. We propose \textbf{Adaptive Curvature-Guided Sparse Zeroth-Order Optimization (CurvZO)}, which tracks curvature signals online from scalar ZO feedback and leverages these signals to construct a parameter-wise sampling distribution for selecting coordinates at each update, reducing the variance of the sparse ZO gradient estimator. Moreover, CurvZO dynamically adapts the perturbation budget to the evolving curvature signal distribution, yielding sparse ZO updates that remain both focused and sufficiently exploratory. Extensive experiments on OPT and Llama across diverse NLP tasks show that CurvZO consistently improves fine-tuning performance and reduces training time over ZO baselines. It improves accuracy by up to 4.4 points and achieves up to a speedup, while preserving memory efficiency.
Paper Structure (39 sections, 8 theorems, 100 equations, 4 figures, 5 tables)

This paper contains 39 sections, 8 theorems, 100 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Zeroth-Order Optimization
  4. Sparsity in ZO
  5. CurvZO
  6. Online Curvature Signal Tracking
  7. Curvature Score.
  8. Stabilizing Curvature Scores.
  9. Curvature-Guided Sparse Zeroth-Order Optimization
  10. Bias Correction for the Gradient Estimator.
  11. Variance-Minimizing Sampling Distribution.
  12. Adaptive Budget Selection
  13. Effective Support Size.
  14. Score Sharpness.
  15. Adaptive Budget Selection.
  16. ...and 24 more sections

Key Result

Proposition 3.2

The gradient estimator defined in Definition def:gra estimator is unbiased, i.e., where $\mathcal{O}(\epsilon^2)$ denotes the standard second-order bias in ZO gradient estimation.

Figures (4)

  • Figure 1: Visualization of anisotropic local curvature in the attention output weights of OPT-6.7B. The $x$- and $y$-axes index the columns and rows of the weight matrix, while the $z$-axis shows curvature magnitude approximated via the diagonal Fisher information (a standard local-curvature surrogate in neural networks).
  • Figure 2: Accuracy (%) on fine-tuning Llama2-7B (top) and Llama2-13B (bottom) with 1,000 training samples.
  • Figure 3: Convergence curves of fine-tuning OPT-2.7B with MeZO and CurvZO on (a) RTE, (b) BoolQ tasks.
  • Figure 4: GPU hours of MeZO and CurvZO across tasks, together with convergence iterations and forward pass time.

Theorems & Definitions (10)

  • Definition 3.1
  • Proposition 3.2: Unbiasedness
  • Proposition 3.3: Variance of the Unbiased Gradient Estimator
  • Proposition 3.4: Variance-Minimizing Sampling Distribution
  • Lemma 3.7: Bounding $\|\nabla \mathcal{L}\|$ by $\|\nabla \mathcal{L}_\pi\|$
  • Theorem 3.8
  • Proposition 3.9: Block-wise Consistency
  • Proposition 3.10: Block-wise Variance-Minimizing Sampling Distribution
  • Lemma 7.1: Approximate Unbiasedness
  • proof