Approximating Opaque Top-k Queries

TL;DR

The paper addresses the challenge of answering top-$k$ queries when the scoring function is an opaque UDF, which can be expensive and evolve over time. It proposes a framework that combines a task-independent hierarchical index (VOODOO) with a histogram-based, $ ext{epsilon}$-greedy DR-submodular bandit to maximize the intrinsic Sum-of-Top-$k$ (STK) objective, providing near-constant-factor guarantees. Theoretical results show monotonicity and DR-submodularity of STK, plus a regret bound that approaches a constant fraction of the optimum as iterations grow; practical enhancements include histogram maintenance, fallback strategies, and batched processing. Extensive experiments on synthetic, tabular, and image data demonstrate substantial speedups over exhaustive scans while achieving near-optimal STK and high Precision@K, validating the method's scalability and applicability to interactive data science workloads with opaque scoring models.

Abstract

Combining query answering and data science workloads has become prevalent. An important class of such workloads is top-k queries with a scoring function implemented as an opaque UDF - a black box whose internal structure and scores on the search domain are unavailable. Some typical examples include costly calls to fuzzy classification and regression models. The models may also be changed in an ad-hoc manner. Since the algorithm does not know the scoring function's behavior on the input data, opaque top-k queries become expensive to evaluate exactly or speed up by indexing. Hence, we propose an approximation algorithm for opaque top-k query answering. Our proposed solution is a task-independent hierarchical index and a novel bandit algorithm. The index clusters elements by some cheap vector representation then builds a tree of the clusters. Our bandit is a diminishing returns submodular epsilon-greedy bandit algorithm that maximizes the sum of the solution set's scores. Our bandit models the distribution of scores in each arm using a histogram, then targets arms with fat tails. We prove that our bandit algorithm approaches a constant factor of the optimal algorithm. We evaluate our standalone library on large synthetic, image, and tabular datasets over a variety of scoring functions. Our method accelerates the time required to achieve nearly optimal scores by up to an order of magnitude compared to exhaustive scan while consistently outperforming baseline sampling algorithms.

Figures (9)

• Figure 1: A high-level overview of our proposed solution during indexing and query execution.
• Figure 2: Visualizing the relative performance of various algorithms. $x$-axis is the number of iterations (i.e. scoring function calls) and the $y$-axis indicates the quality of results in terms of the sum of top-$k$ scores (STK).
• Figure 3: A visualization of our histogram maintenance strategy. (a) Extending the lowest bin as $R_{(k)}$ value grows. (b) Extending the range to contain unexpectedly large scores, and (c) subtracting a histogram (shaded) from another histogram.
• Figure 4: Selecting $100$ highest numbers from synthetic normally distributed data ($n = 50,000$) with 20 clusters. Averaged over 25 runs. (a-b) STK, Precision@K vs time. (c) Ablation study. Note the $y$-axis cutoff.
• Figure 5: Selecting 250 highest valued car listings form the UsedCars dataset ($n = 100,000$) where the valuation is given by an XGBoost model. Averaged over 10 runs. (a-b) STK, Precision@K vs time. (c) End-to-end latency, which includes the index building time and the time to run the algorithm exhaustively over the entire dataset.

Theorems & Definitions (6)

• definition 1: Top-$k$ Bandit Problem
• theorem 1
• theorem 2
• Remark
• corollary 1
• theorem 3