Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Personalized Top-k Set Queries Over Predicted Scores

Sohrab Namazi Nia, Subhodeep Ghosh, Senjuti Basu Roy, Sihem Amer-Yahia

TL;DR

The paper tackles personalized top-$k$ set queries when scoring functions are user-defined and partially computed by expensive external oracles. It introduces a four-task framework that computes score bounds, builds probabilistic models of candidate winners, selects the next oracle query by maximizing entropy reduction, and processes oracle responses to tighten bounds. By decomposing the scoring function into construct-predictable components and using max-convolution for likelihoods, the approach achieves exact top-$k$ results while reducing oracle calls by an order of magnitude. Extensive experiments on multimodal datasets (Hotels, Movies, Yelp) demonstrate both accuracy and scalability, with probabilistic variants offering favorable performance trade-offs. The work significantly advances efficient, oracle-based top-$k$ retrieval in multimodal, personalized settings and opens avenues for extensions to multiple questions, response types, and rank-aware queries.

Abstract

This work studies the applicability of expensive external oracles such as large language models in answering top-k queries over predicted scores. Such scores are incurred by user-defined functions to answer personalized queries over multi-modal data. We propose a generic computational framework that handles arbitrary set-based scoring functions, as long as the functions could be decomposed into constructs, each of which sent to an oracle (in our case an LLM) to predict partial scores. At a given point in time, the framework assumes a set of responses and their partial predicted scores, and it maintains a collection of possible sets that are likely to be the true top-k. Since calling oracles is costly, our framework judiciously identifies the next construct, i.e., the next best question to ask the oracle so as to maximize the likelihood of identifying the true top-k. We present a principled probabilistic model that quantifies that likelihood. We study efficiency opportunities in designing algorithms. We run an evaluation with three large scale datasets, scoring functions, and baselines. Experiments indicate the efficacy of our framework, as it achieves an order of magnitude improvement over baselines in requiring LLM calls while ensuring result accuracy. Scalability experiments further indicate that our framework could be used in large-scale applications.

Personalized Top-k Set Queries Over Predicted Scores

TL;DR

The paper tackles personalized top- set queries when scoring functions are user-defined and partially computed by expensive external oracles. It introduces a four-task framework that computes score bounds, builds probabilistic models of candidate winners, selects the next oracle query by maximizing entropy reduction, and processes oracle responses to tighten bounds. By decomposing the scoring function into construct-predictable components and using max-convolution for likelihoods, the approach achieves exact top- results while reducing oracle calls by an order of magnitude. Extensive experiments on multimodal datasets (Hotels, Movies, Yelp) demonstrate both accuracy and scalability, with probabilistic variants offering favorable performance trade-offs. The work significantly advances efficient, oracle-based top- retrieval in multimodal, personalized settings and opens avenues for extensions to multiple questions, response types, and rank-aware queries.

Abstract

This work studies the applicability of expensive external oracles such as large language models in answering top-k queries over predicted scores. Such scores are incurred by user-defined functions to answer personalized queries over multi-modal data. We propose a generic computational framework that handles arbitrary set-based scoring functions, as long as the functions could be decomposed into constructs, each of which sent to an oracle (in our case an LLM) to predict partial scores. At a given point in time, the framework assumes a set of responses and their partial predicted scores, and it maintains a collection of possible sets that are likely to be the true top-k. Since calling oracles is costly, our framework judiciously identifies the next construct, i.e., the next best question to ask the oracle so as to maximize the likelihood of identifying the true top-k. We present a principled probabilistic model that quantifies that likelihood. We study efficiency opportunities in designing algorithms. We run an evaluation with three large scale datasets, scoring functions, and baselines. Experiments indicate the efficacy of our framework, as it achieves an order of magnitude improvement over baselines in requiring LLM calls while ensuring result accuracy. Scalability experiments further indicate that our framework could be used in large-scale applications.

Paper Structure

This paper contains 23 sections, 3 theorems, 32 equations, 9 figures, 4 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Data Model & Problem Definition
  3. Data Model
  4. Formalism
  5. Proposed Framework
  6. Algorithms
  7. Computing Score Bounds of Candidate Sets
  8. Probabilistic Model for Finding the Answer
  9. Avoiding memory and computational bottleneck
  10. Determining the Next Question
  11. Response Processing
  12. Experimental Evaluation
  13. Experimental Setup
  14. Applications
  15. Implemented Algorithms
  16. ...and 8 more sections

Key Result

lemma 1

ProbInd takes $\Theta(M^2m)$ running time, where $M$ is the number of candidates, and $m$ is the number of discrete values of the pdf designating the score of candidate $c$.

Figures (9)

  • Figure 1: Proposed Framework
  • Figure 2: Computing $\mathbf{P(c_1 \geq c_2)}$
  • Figure 3: #LLM calls varying $k$ - EntrRed using ProbInd vs. Random
  • Figure 4: #LLM calls varying $k$ - EntrRed using ProbDep vs. EntrRed using ProbInd
  • Figure 5: Total time taken for EntrRed using ProbDep vs. EntrRed using ProbInd varying #candidates.
  • ...and 4 more figures

Theorems & Definitions (5)

  • definition 1
  • definition 2
  • lemma 1
  • lemma 2
  • lemma 3