Certificate-Guided Pruning for Stochastic Lipschitz Optimization
Ibne Farabi Shihab, Sanjeda Akter, Anuj Sharma
TL;DR
Certificate-Guided Pruning (CGP) addresses stochastic Lipschitz optimization by maintaining an explicit active set $A_t$ defined via a Lipschitz UCB envelope $U_t$ and a global lower bound $\ell_t$, enabling certifiably suboptimal regions and measurable progress. The paper establishes a shrinkage bound on $\mathrm{Vol}(A_t)$ under a margin condition with near-optimality dimension $\alpha$, achieving a sample complexity of $\tilde{O}(\varepsilon^{-(2+\alpha)})$ and proving minimax optimality up to logarithmic factors. It introduces three extensions—CGP-Adaptive for online learning of $L$, CGP-TR for high dimensions with localized certificates, and CGP-Hybrid that switches to GP refinement when local smoothness is detected—each supported by theory and experiments on 12 benchmarks up to $d=100$. Experiments show CGP variants match or exceed strong baselines and provide principled stopping criteria via certificate volume, with notable speedups over GP-based methods. Overall, CGP delivers a certificate-enabled framework for efficient, reliable black-box optimization with broad practical potential.
Abstract
We study black-box optimization of Lipschitz functions under noisy evaluations. Existing adaptive discretization methods implicitly avoid suboptimal regions but do not provide explicit certificates of optimality or measurable progress guarantees. We introduce \textbf{Certificate-Guided Pruning (CGP)}, which maintains an explicit \emph{active set} $A_t$ of potentially optimal points via confidence-adjusted Lipschitz envelopes. Any point outside $A_t$ is certifiably suboptimal with high probability, and under a margin condition with near-optimality dimension $α$, we prove $\Vol(A_t)$ shrinks at a controlled rate yielding sample complexity $\tildeO(\varepsilon^{-(2+α)})$. We develop three extensions: CGP-Adaptive learns $L$ online with $O(\log T)$ overhead; CGP-TR scales to $d > 50$ via trust regions with local certificates; and CGP-Hybrid switches to GP refinement when local smoothness is detected. Experiments on 12 benchmarks ($d \in [2, 100]$) show CGP variants match or exceed strong baselines while providing principled stopping criteria via certificate volume.