Stochastic Two Points Method for Deep Model Zeroth-order Optimization

TL;DR

This work tackles zeroth-order optimization for training large deep models by introducing Stochastic Two-Point Search (S2P) and its practical variant VS2P. The authors establish convergence guarantees under both general L-smoothness and relaxed (L0,L1)-smoothness, deriving that the query complexity scales as $O\left(\frac{d}{\epsilon^2}\right)$ in both regimes and revealing connections between gradient-free random-search techniques. VS2P enhances practicality via gamma clipping and a sign trick to reduce forward passes, achieving better empirical performance and robustness across image and language tasks, with notable speedups. The results suggest that carefully designed step-size strategies and perturbation-based updates can offer competitive zeroth-order alternatives for adapting deep models without backpropagation, albeit with limitations in lower-bound analyses for certain MeZO-like methods and area for hyper-parameter tuning research.

Abstract

Large foundation models, such as large language models, have performed exceptionally well in various application scenarios. Building or fully fine-tuning such large models is usually prohibitive due to either hardware budget or lack of access to backpropagation. The zeroth-order methods offer a promising direction for tackling this challenge, where only forward passes are needed to update the model. This paper introduces an efficient Stochastic Two-Point (S2P) approach within the gradient-free regime. We present the theoretical convergence properties of S2P under the general and relaxed smoothness assumptions, and the derived results help understand and inherently connect the two popular types of zeroth-order methods, basic random search and stochastic three-point method. The theoretical properties also shed light on a Variant of S2P (VS2P), through exploiting our new convergence properties that better represent the dynamics of deep models in training. Our comprehensive empirical results show that VS2P is highly effective in optimizing objectives for deep models. It outperforms or achieves competitive performance compared to standard methods across various model types and scales.

Figures (9)

• Figure 1: Fully fine-tuning DenseNet121, ResNet18, ViT-B16, and VGG11 with proposed method and baselines under datasets MNIST and CIFAR10 & 200 training epochs.
• Figure 2: Fine-tuning GPT2 under selected tasks of GLUE benchmark experiments. The selected learning rate is highlighted in line labels.
• Figure 3: Fully fine-tuning DenseNet121, ResNet18, ViT-B16, and VGG11 under various learning rates & datasets MNIST & 200 training epochs & 3 different seeds.
• Figure 4: Fully fine-tuning DenseNet121, ResNet18, ViT-B16, and VGG11 under various learning rates & datasets CIFAR10 & 200 training epochs & 3 different seeds.
• Figure 5: Fully fine-tuning DenseNet121, ResNet18, ViT-B16, and VGG11 with proposed method and baselines under datasets MNIST and CIFAR10 & 100 epochs & 3 different seeds.

Theorems & Definitions (22)

• Definition 3.1
• Definition 3.2
• Definition 3.3
• Lemma 3.6
• Lemma 3.7: Progressive bound
• Theorem 3.8: Query complexity
• Remark 3.9: Inherent connection between S2P and MeZO
• Lemma 3.10: Progressive bound
• Theorem 3.11: Query complexity
• Remark 3.12: Step size constrain