iPDB -- Optimizing SQL Queries with ML and LLM Predicates

TL;DR

iPDB presents a declarative extension to SQL that embeds in-database ML and LLM inference via a native PREDICT operator and semantic predicates. It builds on DuckDB, introducing a model catalog, extended parsing/planning/execution, and a range of optimizations (prompt deduplication, multi-row marshaling, parallelization) to dramatically reduce latency and token usage while preserving accuracy. The approach enables in-database semantic queries such as semantic selects, joins, projections, and aggregates, avoiding data movement and external inference pipelines. Empirical results across multiple datasets and models demonstrate up to 1000x improvements in latency and substantial cost and accuracy benefits, underscoring the practical impact of tightly integrating ML/LLMs with relational query processing.

Abstract

Structured Query Language (SQL) has remained the standard query language for databases. SQL is highly optimized for processing structured data laid out in relations. Meanwhile, in the present application development landscape, it is highly desirable to utilize the power of learned models to perform complex tasks. Large language models (LLMs) have been shown to understand and extract information from unstructured textual data. However, SQL as a query language and accompanying relational database systems are either incompatible or inefficient for workloads that require leveraging learned models. This results in complex engineering and multiple data migration operations that move data between the data sources and the model inference platform. In this paper, we present iPDB, a relational system that supports in-database machine learning (ML) and large language model (LLM) inferencing using extended SQL syntax. In iPDB, LLMs and ML calls can function as semantic projects, as predicates to perform semantic selects and semantic joins, or for semantic grouping in group-by clauses. iPDB has a novel relational predict operator and semantic query optimizations that enable users to write and efficiently execute semantic SQL queries, outperforming the state-of-the-art.

Figures (7)

• Figure 1: High-level architecture of iPDB
• Figure 2: Query Execution Process in iPDB to support semantic operators
• Figure 3: Performance analysis of intra-operator optimizations under sequential and parallel (16 workers) execution.
• Figure 4: Generation latency vs row-marshaled batch size
• Figure 5: Query Latency of row-marshaling vs parallelization