Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Noisy Pairwise-Comparison Random Search for Smooth Nonconvex Optimization

Taha El Bakkali, Rayane Bouftini, Qiuyi Zhang, Omar Saadi

TL;DR

The paper tackles optimization of smooth nonconvex objectives using only noisy pairwise comparisons, addressing the mismatch between ambient and intrinsic problem dimensionality. It introduces Noisy-Comparison Random Search (NCRS), a direct-search method whose updates effectively occur in the low-dimensional active subspace, yielding an $\mathcal{O}(k/(p^{2}\varepsilon^{2}))$ complexity under a uniform-margin oracle. It further extends to a gap-aware confidence model with a majority-vote variant that attains $\mathcal{O}(k^{2}/\varepsilon^{4})$ queries, incorporating near-tie uncertainty. Theoretical results are complemented by experiments showing intrinsic-dimension adaptation in language-model fine-tuning and competitive performance on preference-based RL tasks, highlighting practical impact for high-dimensional zeroth-order optimization with limited feedback.

Abstract

We consider minimizing high-dimensional smooth nonconvex objectives using only noisy pairwise comparisons. Unlike classical zeroth-order methods limited by the ambient dimension $d$, we propose Noisy-Comparison Random Search (NCRS), a direct-search method that exploits random line search to adapt to the intrinsic dimension $k \le d$. We establish novel non-convex convergence guarantees for approximate stationarity: under a uniform-margin oracle, NCRS attains $ε$-stationarity with complexity $\mathcal{O}(k/(p^{2}ε^{2}))$, explicitly replacing ambient dependence with the intrinsic dimension. Furthermore, we introduce a general tie-aware noise model where comparison quality degrades near ties; for this setting, we prove that a majority-vote variant of NCRS achieves $ε$-stationarity with complexity $\mathcal{O}(k^{2}/ε^{4})$.

Noisy Pairwise-Comparison Random Search for Smooth Nonconvex Optimization

TL;DR

The paper tackles optimization of smooth nonconvex objectives using only noisy pairwise comparisons, addressing the mismatch between ambient and intrinsic problem dimensionality. It introduces Noisy-Comparison Random Search (NCRS), a direct-search method whose updates effectively occur in the low-dimensional active subspace, yielding an complexity under a uniform-margin oracle. It further extends to a gap-aware confidence model with a majority-vote variant that attains queries, incorporating near-tie uncertainty. Theoretical results are complemented by experiments showing intrinsic-dimension adaptation in language-model fine-tuning and competitive performance on preference-based RL tasks, highlighting practical impact for high-dimensional zeroth-order optimization with limited feedback.

Abstract

We consider minimizing high-dimensional smooth nonconvex objectives using only noisy pairwise comparisons. Unlike classical zeroth-order methods limited by the ambient dimension , we propose Noisy-Comparison Random Search (NCRS), a direct-search method that exploits random line search to adapt to the intrinsic dimension . We establish novel non-convex convergence guarantees for approximate stationarity: under a uniform-margin oracle, NCRS attains -stationarity with complexity , explicitly replacing ambient dependence with the intrinsic dimension. Furthermore, we introduce a general tie-aware noise model where comparison quality degrades near ties; for this setting, we prove that a majority-vote variant of NCRS achieves -stationarity with complexity .
Paper Structure (27 sections, 24 theorems, 137 equations, 5 figures, 4 tables, 3 algorithms)

This paper contains 27 sections, 24 theorems, 137 equations, 5 figures, 4 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Intrinsic dimension via an active--nuisance subspace model.
  4. Stochastic sign-comparison oracle.
  5. Our contributions
  6. Uniform-margin ranking oracle: NCRS analysis
  7. Convergence analysis of NCRS for ridge objectives $f(x)=g(Ax)$
  8. Beyond exact ridge: NCRS under weak inactive dependence
  9. Gap-dependent confidence oracle: NCRS analysis with confidence-weighted vote
  10. Numerical Experiments
  11. Intrinsic Dimension Effect
  12. Preference-Based Reinforcement Learning
  13. Preliminary
  14. Hyperparameters
  15. LLM Setting
  16. ...and 12 more sections

Key Result

Lemma 2.1

Assume that $f(x)=g(Ax)$ for some matrix $A\in\mathbb{R}^{k\times d}$ with full row rank $\mathop{\mathrm{rank}}\nolimits(A)=k\le d$ and some function $g:\mathbb{R}^k\to\mathbb{R}$, and that $f$ is $L_f$-smooth. Let $(\theta^t)$ be the iterates produced by alg:NCRS-sto, and suppose the ranking oracl

Figures (5)

  • Figure 1: Zeroth-order methods adapt to intrinsic dimension. Number of iterations required to reach a fixed target accuracy across different language models fine-tuning settings with decreasing intrinsic dimension. Shaded areas represent the 95% CI over 5 runs.
  • Figure 2: Environments. We evaluate preference-based reinforcement learning algorithms on six locomotion tasks.
  • Figure 3: Comparison between preference learning algorithms on DM Control Suite tasks. Results show the mean ground truth reward and standard error (y-axis) over 600 episodes (x-axis) for 5 random seeds where each algorithm collects 64 trajectories per episode.
  • Figure 4: Learning rate sweep. Validation loss for different models on the MRPC task for SGD and NCRS.
  • Figure 5: 2D parameter sweep. Validation loss for different models on the MRPC task for RSGF with varying learning rate $\alpha$ and perturbation $\mu$.

Theorems & Definitions (42)

  • Remark 1.1: Why Assumption \ref{['assump:signal-variance-score']} is more realistic than uniform margin
  • Lemma 2.1
  • Theorem 2.2
  • Remark 2.3
  • Proposition 2.4
  • Remark 2.5
  • Lemma 3.1
  • Lemma 3.2
  • Lemma 3.3
  • Proposition 3.4
  • ...and 32 more