Stochastic Subspace Descent Accelerated via Bi-fidelity Line Search

TL;DR

This paper tackles the challenge of expensive function and gradient evaluations in high-dimensional, black-box optimization. It introduces BF-SSD, a zeroth-order method that builds a local bi-fidelity surrogate combining HF and LF evaluations to perform an efficient Armijo backtracking line search along a stochastic subspace; this reduces reliance on HF calls while preserving convergence guarantees. The authors establish convergence results under standard smoothness assumptions and finite-sample surrogate accuracy, and validate BF-SSD across synthetic benchmarks, kernel ridge regression, black-box adversarial attacks, and soft prompting for language models. The results show BF-SSD consistently achieves superior optimization performance with substantially fewer HF evaluations, highlighting the practical impact of bi-fidelity strategies for large-scale, high-dimensional problems.

Abstract

Efficient optimization remains a fundamental challenge across numerous scientific and engineering domains, especially when objective function and gradient evaluations are computationally expensive. While zeroth-order optimization methods offer effective approaches when gradients are inaccessible, their practical performance can be limited by the high cost associated with function queries. This work introduces the bi-fidelity stochastic subspace descent (BF-SSD) algorithm, a novel zeroth-order optimization method designed to reduce this computational burden. BF-SSD leverages a bi-fidelity framework, constructing a surrogate model from a combination of computationally inexpensive low-fidelity (LF) and accurate high-fidelity (HF) function evaluations. This surrogate model facilitates an efficient backtracking line search for step size selection, for which we provide theoretical convergence guarantees under standard assumptions. We perform a comprehensive empirical evaluation of BF-SSD across four distinct problems: a synthetic optimization benchmark, dual-form kernel ridge regression, black-box adversarial attacks on machine learning models, and transformer-based black-box language model fine-tuning. Numerical results demonstrate that BF-SSD consistently achieves superior optimization performance while requiring significantly fewer HF function evaluations compared to relevant baseline methods. This study highlights the efficacy of integrating bi-fidelity strategies within zeroth-order optimization, positioning BF-SSD as a promising and computationally efficient approach for tackling large-scale, high-dimensional problems encountered in various real-world applications.

Figures (24)

• Figure 1: Gradient descent (GD), coordinate descent (CD), and stochastic subspace descent (SSD), along with their respective backtracking line search (LS) variants for step size tuning, as well as the proposed Bi-fidelity SSD (BF-SSD), are evaluated on the "worst function in the world" example, detailed in Section \ref{['ssec:worst-function']}.
• Figure 2: Illustration of the bi-fidelity backtracking line search process using the example problem in Section \ref{['ssec:kernel-ridge']}. The blue curve represents the bi-fidelity surrogate model $\tilde{\varphi}_k$ approximating the HF function $\varphi$ (red curve). It significantly lowers computational cost (e.g., reducing 4 HF calls to 1 HF + 6 LF calls).
• Figure 3:
• Figure 4:
• Figure 5:

Theorems & Definitions (15)

• Theorem 2.4
• Remark 2.5
• Remark 2.6
• Lemma 2.7
• proof
• Remark 2.8
• Remark 2.9
• Theorem 3.1: proof in Appendix \ref{['sec:B3']}
• proof
• Theorem B.2