Explicit and Non-asymptotic Query Complexities of Rank-Based Zeroth-order Algorithm on Stochastic Smooth Functions
Haishan Ye
TL;DR
This paper studies rank-based zeroth-order optimization for stochastic smooth functions where only ordinal feedback is available. It introduces SA1, a simple rank-based ZO algorithm that uses Gaussian direction sampling and a rank oracle to form a descent direction without evaluating exact function values, and provides explicit non-asymptotic query complexities. Theoretical results show that, under standard assumptions, ordinal feedback yields the same order of query efficiency as value-based ZO methods: $\mathcal{O}(\frac{dLG_u^2}{\mu^2\varepsilon})$ for strongly convex objectives and $\mathcal{O}(\frac{dLG_u^2}{\varepsilon^2})$ for nonconvex objectives. The analysis introduces a novel event-based framework, avoiding drift and information-geometric techniques, and demonstrates robustness of rank-based methods to stochastic noise while achieving optimal sample efficiency.
Abstract
Zeroth-order (ZO) optimization with ordinal feedback has emerged as a fundamental problem in modern machine learning systems, particularly in human-in-the-loop settings such as reinforcement learning from human feedback, preference learning, and evolutionary strategies. While rank-based ZO algorithms enjoy strong empirical success and robustness properties, their theoretical understanding, especially under stochastic objectives and standard smoothness assumptions, remains limited. In this paper, we study rank-based zeroth-order optimization for stochastic functions where only ordinal feedback of the stochastic function is available. We propose a simple and computationally efficient rank-based ZO algorithm. Under standard assumptions including smoothness, strong convexity, and bounded second moments of stochastic gradients, we establish explicit non-asymptotic query complexity bounds for both convex and nonconvex objectives. Notably, our results match the best-known query complexities of value-based ZO algorithms, demonstrating that ordinal information alone is sufficient for optimal query efficiency in stochastic settings. Our analysis departs from existing drift-based and information-geometric techniques, offering new tools for the study of rank-based optimization under noise. These findings narrow the gap between theory and practice and provide a principled foundation for optimization driven by human preferences.