Beyond Relational: Semantic-Aware Multi-Modal Analytics with LLM-Native Query Optimization

TL;DR

Nirvana tackles the challenge of semantic-aware multi-modal analytics by decoupling semantic processing from traditional relational operators and introducing two novel optimizers. The agentic logical plan optimizer uses LLM-guided rewriting and random-walk search, with LLM-as-a-judge to assess plan quality, while the physical plan optimizer selects cost-effective LLM backends per operator using an improvement-score metric, aided by computation reuse and evaluation pushdown. Together, these components yield substantial end-to-end improvements in runtime and monetary cost across three real-world benchmarks, with competitive or superior result quality compared to state-of-the-art semantic analytics systems. The work demonstrates scalable, cost-efficient semantic analytics over multi-modal data and provides open-source code for broader adoption and further optimization research.

Abstract

Multi-modal analytical processing has the potential to transform applications in e-commerce, healthcare, entertainment, and beyond. However, real-world adoption remains elusive due to the limited ability of traditional relational query operators to capture query semantics. The emergence of foundation models, particularly the large language models (LLMs), opens up new opportunities to develop flexible, semantic-aware data analytics systems that transcend the relational paradigm. We present Nirvana, a multi-modal data analytics framework that incorporates programmable semantic operators while leveraging both logical and physical query optimization strategies, tailored for LLM-driven semantic query processing. Nirvana addresses two key challenges. First, it features an agentic logical optimizer that uses natural language-specified transformation rules and random-walk-based search to explore vast spaces of semantically equivalent query plans -- far beyond the capabilities of conventional optimizers. Second, it introduces a cost-aware physical optimizer that selects the most effective LLM backend for each operator using a novel improvement-score metric. To further enhance efficiency, Nirvana incorporates computation reuse and evaluation pushdown techniques guided by model capability hypotheses. Experimental evaluations on three real-world benchmarks demonstrate that Nirvana is able to reduce end-to-end runtime by 10%--85% and reduces system processing costs by 76% on average, outperforming state-of-the-art systems at both efficiency and scalability.

Figures (15)

• Figure 1: Example of performing data analytics on a multi-modal movie table by (1) directly prompting an LLM or (2) using a data analytical processing system.
• Figure 2: System overview of Nirvana. Given an initial query plan, the logical optimizer samples and rewrites the logical plan to identify a (near-)optimal plan in the logical plan optimization phase. Then, the physical optimizer assigns the most cost-effective model to each operator in the logical plan.
• Figure 3: A running example of logical plan optimization where, each node (a logical plan) is sampled by random walk.
• Figure 4: Overview of physical plan optimization.
• Figure 5: Possible responses of different models to a query, based on the model capability hypothesis. $m^{*}(x)$ denotes the response of the most powerful model $m^{*}$ to the input $x$, while $\overline{m^{*}(x)}$ denotes the opposite response to the same input.

Theorems & Definitions (1)

• Example 1: Prompt vs. Compound System