Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SharpZO: Hybrid Sharpness-Aware Vision Language Model Prompt Tuning via Forward-Only Passes

Yifan Yang, Zhen Zhang, Rupak Vignesh Swaminathan, Jing Liu, Nathan Susanj, Zheng Zhang

TL;DR

SharpZO proposes a forward-only, two-stage fine-tuning framework for vision-language models that eschews backpropagation. Stage 1 uses a sharpness-aware CMA-ES warm-up to globally smooth the loss landscape and provide a strong initialization, while Stage 2 performs a sparse zeroth-order local search with a novel Z-pruning scheme to reduce variance and search dimensionality. Theoretical analysis under PL and Lipschitz assumptions yields a convergence rate showing the benefit of sharpness-aware initialization, and extensive experiments on CLIP demonstrate superior accuracy and efficiency over existing BP-free baselines, including robustness to distribution shifts. Overall, SharpZO offers a practical, memory-efficient alternative for edge-device deployment of VLM prompt-tuning with competitive performance gains.

Abstract

Fine-tuning vision language models (VLMs) has achieved remarkable performance across various downstream tasks; yet, it requires access to model gradients through backpropagation (BP), making them unsuitable for memory-constrained, inference-only edge devices. To address this limitation, previous work has explored various BP-free fine-tuning methods. However, these approaches often rely on high-variance evolutionary strategies (ES) or zeroth-order (ZO) optimization, and often fail to achieve satisfactory performance. In this paper, we propose a hybrid Sharpness-aware Zeroth-order optimization (SharpZO) approach, specifically designed to enhance the performance of ZO VLM fine-tuning via a sharpness-aware warm-up training. SharpZO features a two-stage optimization process: a sharpness-aware ES stage that globally explores and smooths the loss landscape to construct a strong initialization, followed by a fine-grained local search via sparse ZO optimization. The entire optimization relies solely on forward passes. Detailed theoretical analysis and extensive experiments on CLIP models demonstrate that SharpZO significantly improves accuracy and convergence speed, achieving up to 7% average gain over state-of-the-art forward-only methods.

SharpZO: Hybrid Sharpness-Aware Vision Language Model Prompt Tuning via Forward-Only Passes

TL;DR

SharpZO proposes a forward-only, two-stage fine-tuning framework for vision-language models that eschews backpropagation. Stage 1 uses a sharpness-aware CMA-ES warm-up to globally smooth the loss landscape and provide a strong initialization, while Stage 2 performs a sparse zeroth-order local search with a novel Z-pruning scheme to reduce variance and search dimensionality. Theoretical analysis under PL and Lipschitz assumptions yields a convergence rate showing the benefit of sharpness-aware initialization, and extensive experiments on CLIP demonstrate superior accuracy and efficiency over existing BP-free baselines, including robustness to distribution shifts. Overall, SharpZO offers a practical, memory-efficient alternative for edge-device deployment of VLM prompt-tuning with competitive performance gains.

Abstract

Fine-tuning vision language models (VLMs) has achieved remarkable performance across various downstream tasks; yet, it requires access to model gradients through backpropagation (BP), making them unsuitable for memory-constrained, inference-only edge devices. To address this limitation, previous work has explored various BP-free fine-tuning methods. However, these approaches often rely on high-variance evolutionary strategies (ES) or zeroth-order (ZO) optimization, and often fail to achieve satisfactory performance. In this paper, we propose a hybrid Sharpness-aware Zeroth-order optimization (SharpZO) approach, specifically designed to enhance the performance of ZO VLM fine-tuning via a sharpness-aware warm-up training. SharpZO features a two-stage optimization process: a sharpness-aware ES stage that globally explores and smooths the loss landscape to construct a strong initialization, followed by a fine-grained local search via sparse ZO optimization. The entire optimization relies solely on forward passes. Detailed theoretical analysis and extensive experiments on CLIP models demonstrate that SharpZO significantly improves accuracy and convergence speed, achieving up to 7% average gain over state-of-the-art forward-only methods.

Paper Structure

This paper contains 27 sections, 3 theorems, 41 equations, 4 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Coordinate-wise Gradient Estimation or Randomized Gradient Estimation?
  4. Covariance Matrix Adaptation Evolution Strategy (CMA-ES)
  5. Sharpness-aware Optimization
  6. The SharpZO Method
  7. Stage 1: Sharpness-Aware CMA-ES Method
  8. Stage 2: Fine-grained ZO Method
  9. Algorithms
  10. Theoretical Guarantee
  11. Experiments
  12. Results on Few-Shot Classification
  13. Robustness to Distribution Shift
  14. Time-to-test-accuracy Efficiency
  15. Ablation on Components in Different Stages
  16. ...and 12 more sections

Key Result

Theorem 1

Under assumptions A1 and A2, suppose the SharpZO algorithm first performs $T_c$ steps of global optimization using CMA-ES and then switches to zeroth-order gradient-based optimization until convergence. The convergence rate of SharpZO method can be give by: where $\epsilon$ is given by assuming $\mathcal{L}(\bm{w}_t) - \mathcal{L}^* \le \epsilon$, $\eta$ is the learning rate of ZO-SGD optimizer i

Figures (4)

  • Figure 1: (a) Comparison between SharpZO and other ZO prompt-tuning baselines.SharpZO demonstrates significantly lower variance than other ZO-based baselines like ZIP park2025zip and BlackVIP oh2023blackvip. (b) Fine-tuned performance across all 11 tasks tested compared with ZIP and BlackVIP and BBT sun2022black. All experiments are conducted using the CLIP model with a ViT-B/16 backbone.
  • Figure 2: Overview of the SharpZO method. (a) The overall training pipeline of SharpZO, consisting of a two-stage optimization process. (b) Visualization of the smoothed loss landscape after Stage 1 sharpness-aware CMA-ES optimization. (c) Training dynamics of the sharpness-aware CMA-ES method. (d) RGE-based gradient estimation during sparse ZO training in Stage 2.
  • Figure 3: Comparison between the naive CMA-ES with the Sharpness-aware (S-aware) CMA-ES method on EuroSAT dataset.
  • Figure 4: Comparison between the naive ZO optimization and sparse ZO optimization with various pruning metrics on EuroSAT dataset.

Theorems & Definitions (6)

  • Theorem 1
  • proof
  • Lemma 1: Per–step error bound for sharpness-aware CMA-ES
  • proof
  • Lemma 2: Per-step Error Bound for ZO-SGD
  • proof