Top-k on a Budget: Adaptive Ranking with Weak and Strong Oracles

TL;DR

The paper tackles certifying the exact top-$k$ set under a two-oracle model with a fast, noisy weak oracle and a scarce, high-fidelity strong oracle. It introduces a screen-then-certify baseline (STC) with a strong-call bound tied to the near-tie mass $m(\cdot)$ and proves a matching conditional lower bound, establishing the intrinsic difficulty. The authors then propose ACE, an adaptive certification algorithm that concentrates strong evaluations on border items to achieve the same $O(m(4\varepsilon_{\max}))$ bound while reducing practical strong calls, and ACE-W, a two-phase method that first allocates the weak budget adaptively to shrink ambiguity before certification. The results show substantial reductions in strong calls in both synthetic and real-data valuation tasks, with performance governed by the near-tie mass and weak oracle quality. Overall, the work provides instance-dependent PAC guarantees and practical adaptive strategies for efficient top-$k$ certification with heterogeneous information sources.

Abstract

Identifying the top-$k$ items is fundamental but often prohibitive when exact valuations are expensive. We study a two-oracle setting with a fast, noisy weak oracle and a scarce, high-fidelity strong oracle (e.g., human expert verification or expensive simulation). We first analyze a simple screen-then-certify baseline (STC) and prove it makes at most $m(4\varepsilon_{\max})$ strong calls given jointly valid weak confidence intervals with maximum radius $\varepsilon_{\max}$, where $m(\cdot)$ denotes the near-tie mass around the top-$k$ threshold. We establish a conditional lower bound of $Ω(m(\varepsilon_{\max}))$ for any algorithm given the same weak uncertainty. Our main contribution is ACE, an adaptive certification algorithm that focuses strong queries on critical boundary items, achieving the same $O(m(4\varepsilon_{\max}))$ bound while reducing strong calls in practice. We then introduce ACE-W, a fully adaptive two-phase method that allocates weak budget adaptively before running ACE, further reducing strong costs.

Figures (1)

• Figure 1: Performance on synthetic data. Our adaptive ACE and ACE-W algorithms consistently outperform the non-adaptive baselines.

Theorems & Definitions (17)

• lemma 1
• proof
• theorem 1
• proof
• theorem 2
• proof
• theorem 3
• proof
• theorem 4
• proof