Conformal Prediction for Verifiable Learned Query Optimization

TL;DR

This work tackles the reliability issues of Learned Query Optimizers by introducing Conformal Prediction (CP) to provide verifiable latency guarantees. It develops CP-based latency bounds for both static and distribution-shift scenarios, enables CP-driven runtime verification during plan construction, and proposes an adaptive CP framework to maintain confidence under workload changes. A CP-guided plan-search mechanism further improves plan quality and planning time, demonstrated across three LQOs (Balsa, Lero, RTOS) and multiple benchmarks (JOB, TPC-H, JOBLight, CEB). The results show tight latency bounds, effective violation detection and handling, robustness to distribution shifts, and notable planning-time and plan-quality gains, indicating CP as a practical, scalable verification tool for learned database components.

Abstract

Query optimization is critical in relational databases. Recently, numerous Learned Query Optimizers (LQOs) have been proposed, demonstrating superior performance over traditional hand-crafted query optimizers after short training periods. However, the opacity and instability of machine learning models have limited their practical applications. To address this issue, we are the first to formulate the LQO verification as a Conformal Prediction (CP) problem. We first construct the CP model and obtain user-controlled bounded ranges for the actual latency of LQO plans before execution. Then, we introduce CP-based runtime verification along with violation handling to ensure performance prior to execution. For both scenarios, we further extend our framework to handle distribution shifts in the dynamic environment using adaptive CP approaches. Finally, we present CP-guided plan search, which uses actual latency upper bounds from CP to heuristically guide query plan construction. We integrated our verification framework into three LQOs (Balsa, Lero, and RTOS) and conducted evaluations on the JOB and TPC-H workloads. Experimental results demonstrate that our method is both accurate and efficient. Our CP-based approaches achieve tight upper bounds, reliably detect and handle violations. Adaptive CP maintains accurate confidence levels even in the presence of distribution shifts, and the CP-guided plan search improves both query plan quality (up to 9.84x) and planning time, with a reduction of up to 74.4% for a single query and 9.96% across all test queries from trained LQOs.

Figures (16)

• Figure 1: ML Decisions in Learned Query Optimizers (LQOs).
• Figure 2: CP-based Bounded Latency Range Framework.
• Figure 3: Bounded Latency Ranges with Different Types of Upper Bound ($C$).
• Figure 4: CP-based Runtime Verification Framework.
• Figure 5: Unified-cased Upper Bounds.