Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Access Paths for Efficient Ordering with Large Language Models

Fuheng Zhao, Jiayue Chen, Yiming Pan, Tahseen Rabbani, Sohaib, Divyakant Agrawal, Amr El Abbadi, Paritosh Aggarwal, Anupam Datta, Dimitris Tsirogiannis

TL;DR

This work investigates how to implement LLM ORDER BY for semantic sorting in large-scale analytic settings. It shows that no single algorithm universally dominates across tasks, motivating an adaptive optimizer that combines cost estimation, world-knowledge signals, and consensus-based strategies (Borda Count and LLM-as-Judge) to select near-optimal access paths. The authors introduce new algorithms, including a semantic external merge sort, and demonstrate that their adaptive approach consistently matches or exceeds static baselines across diverse datasets while respecting budgets. The results establish a practical, cost-aware framework for robust LLM-powered sorting in analytics pipelines and highlight a test-time scaling relationship between compute and accuracy.

Abstract

In this work, we present the \texttt{LLM ORDER BY} semantic operator as a logical abstraction and conduct a systematic study of its physical implementations. First, we propose several improvements to existing semantic sorting algorithms and introduce a semantic-aware external merge sort algorithm. Our extensive evaluation reveals that no single implementation offers universal optimality on all datasets. From our evaluations, we observe a general test-time scaling relationship between sorting cost and the ordering quality for comparison-based algorithms. Building on these insights, we design a budget-aware optimizer that utilizes heuristic rules, LLM-as-Judge evaluation, and consensus aggregation to dynamically select the near-optimal access path for LLM ORDER BY. In our extensive evaluations, our optimizer consistently achieves ranking accuracy on par with or superior to the best static methods across all benchmarks. We believe that this work provides foundational insights into the principled optimization of semantic operators essential for building robust, large-scale LLM-powered analytic systems.

Access Paths for Efficient Ordering with Large Language Models

TL;DR

This work investigates how to implement LLM ORDER BY for semantic sorting in large-scale analytic settings. It shows that no single algorithm universally dominates across tasks, motivating an adaptive optimizer that combines cost estimation, world-knowledge signals, and consensus-based strategies (Borda Count and LLM-as-Judge) to select near-optimal access paths. The authors introduce new algorithms, including a semantic external merge sort, and demonstrate that their adaptive approach consistently matches or exceeds static baselines across diverse datasets while respecting budgets. The results establish a practical, cost-aware framework for robust LLM-powered sorting in analytics pipelines and highlight a test-time scaling relationship between compute and accuracy.

Abstract

In this work, we present the \texttt{LLM ORDER BY} semantic operator as a logical abstraction and conduct a systematic study of its physical implementations. First, we propose several improvements to existing semantic sorting algorithms and introduce a semantic-aware external merge sort algorithm. Our extensive evaluation reveals that no single implementation offers universal optimality on all datasets. From our evaluations, we observe a general test-time scaling relationship between sorting cost and the ordering quality for comparison-based algorithms. Building on these insights, we design a budget-aware optimizer that utilizes heuristic rules, LLM-as-Judge evaluation, and consensus aggregation to dynamically select the near-optimal access path for LLM ORDER BY. In our extensive evaluations, our optimizer consistently achieves ranking accuracy on par with or superior to the best static methods across all benchmarks. We believe that this work provides foundational insights into the principled optimization of semantic operators essential for building robust, large-scale LLM-powered analytic systems.

Paper Structure

This paper contains 23 sections, 5 figures, 3 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Methodology
  4. Value-Based Ordering
  5. Comparison-Based Ordering
  6. Limit K
  7. Initial Evaluation
  8. Implementation and Metrics
  9. Observations
  10. Optimizer Design: LLM-Guided and Pessimistic Strategies
  11. Cost Estimation
  12. World Knowledge
  13. Candidate Algorithms
  14. LLM-as-Judge Optimizer
  15. Pessimistic Optimizer
  16. ...and 8 more sections

Figures (5)

  • Figure 1: Sorting accuracy vs. the monetary budget ($). While more budget generally leads to better accuracy, these figures often display a log-linear growth pattern (red curve), with accuracy increasing quickly at first before plateauing.
  • Figure 2: Accuracy distribution across algorithms for the LLaMA 3.1 70B model. Each dot represents a query; star markers denote the mean accuracy, and bars indicate the median.
  • Figure 3: Sorting accuracy vs. the monetary budget ($).
  • Figure 4: Llama models' accuracy distribution on the DL20 across different algorithms and our proposed optimizers.
  • Figure 5: Limiting the optimizer budget to $6 and $70.

Theorems & Definitions (4)

  • Example 1.1
  • Example 1.2
  • Example 5.1
  • Example 5.2