ZOBA: An Efficient Single-loop Zeroth-order Bilevel Optimization Algorithm

TL;DR

This work tackles black-box bilevel optimization where only function evaluations are available by introducing ZOBA, the first single-loop zeroth-order bilevel algorithm that uses delayed information to construct parallelizable hypergradient surrogates. It also offers HF-ZOBA, a Hessian-free variant that avoids explicit Hessian computations. Theoretical results show non-convex convergence with complexity $\mathcal{O}(p(d+p)^2\varepsilon^{-2})$ for ZOBA and $\mathcal{O}((d+p)\varepsilon^{-2})$ for HF-ZOBA, improving over prior two-loop zeroth-order methods. Empirical results on synthetic and real-world adversarial tasks demonstrate competitive accuracy with substantially faster runtimes due to parallelization and evaluation reuse.

Abstract

Bilevel optimization problems consist of minimizing a value function whose evaluation depends on the solution of an inner optimization problem. These problems are typically tackled using first-order methods that require computing the gradient of the value function ({\it the hypergradient}). In several practical settings, however, first-order information is unavailable ({\it zeroth-order setting}), rendering these methods inapplicable. Finite-difference methods provide an alternative by approximating hypergradients using function evaluations along a set of directions. Nevertheless, such surrogates are notoriously expensive, and existing finite-difference bilevel methods rely on two-loop algorithms that are poorly parallelizable. In this work, we propose ZOBA, the first finite-difference single-loop algorithm for bilevel optimization. Our method leverages finite-difference hypergradient approximations based on delayed information to eliminate the need for nested loops. We analyze the proposed algorithm and establish convergence rates in the non-convex setting, achieving a complexity of $\mathcal{O}(p(d + p)^2\varepsilon^{-2})$, where $p$ and $d$ denote the dimension of inner and outer spaces respectively, which is better than prior approaches based on Hessian approximation. We further introduce and analyze HF-ZOBA, a Hessian-free variant that yields additional complexity improvements. Finally, we corroborate our findings with numerical experiments on synthetic functions and a real-world black-box task in adversarial machine learning. Our results show that our methods achieve accuracy comparable to state-of-the-art techniques while requiring less computation time.

Figures (5)

• Figure 1: Comparison of algorithms on synthetic bilevel problems with varying dimensions: objective value versus function evaluations (first row) and versus wall-clock time in seconds (second row).
• Figure 2: Comparison of algorithms on minimal distortion universal perturbation attack. Normalized outer function values and inner function values on iterations $z_k,x_k$ and classification accuracy are reported as function of number of function evaluations (first row) and versus wall-clock time in seconds (second row).
• Figure 3: Normalized objective value progress at the final iterate obtained by running Algorithm \ref{['alg:zoba']} with different stepsizes $\rho$, numbers of directions $\ell$, and batch sizes $b$. If the algorithm diverges, the quantity $(\Psi(x_K) - \min \Psi)/(\Psi(x_0) - \min \Psi)$ is clipped to one. In the first row, every plot show how this quantity change for different values of $\ell$ by fixing the batch size $b$; in the second row, instead, every plot show how such value change for different values of $b$ by fixing $\ell$.
• Figure 4: Normalized objective value progress at the final iterate obtained by running Algorithm \ref{['alg:hfzoba']} with $\gamma = \rho/5$ for different stepsizes $\rho$, numbers of directions $\ell$, and batch sizes $b$. If the algorithm diverges, the quantity $(\Psi(x_K) - \min \Psi)/(\Psi(x_0) - \min \Psi)$ is clipped to one. In the first row, every plot show how this quantity change for different values of $\ell$ by fixing the batch size $b$; in the second row, instead, every plot show how such value change for different values of $b$ by fixing $\ell$.
• Figure 5: Comparison of algorithms on minimal-distortion universal perturbation attacks across different test sets with different labels $y$. For each test set, we report the normalized outer function values, inner function values at iterations $z_k, x_k$, and classification accuracy. The top row of each block shows these metrics as a function of the number of function evaluations, while the bottom row shows them versus wall-clock time (in seconds).

Theorems & Definitions (41)

• Theorem 3.4: Convergence Rate of ZOBA
• Corollary 3.5
• Theorem 3.6: Convergence Rate of HF-ZOBA
• Corollary 3.7
• Lemma C.1
• proof
• Lemma C.2: Regularity of $\Psi$, $z^*$ and $v^*$
• proof
• Lemma C.3: Bound on norm of $v^*$
• proof