Query Optimization in the Wild: Realities and Trends
Yuanyuan Tian
TL;DR
The paper argues that traditional cost-based query optimization, rooted in Cascades/Volcano/System R frameworks, remains pervasive but is poorly suited to modern cloud-scale, multi-engine environments. It identifies three pragmatic reforms: tighter collaboration between query optimization and execution to tolerate suboptimal plans, convergence of optimization with workload optimization to leverage historical and workload-level information, and a shift from monolithic to composable QO architectures that enable cross-engine collaboration and QO-as-a-service concepts. It supports these claims with industrial history (e.g., Microsoft’s optimizer lineage), evidence of cost-model brittleness and cardinality estimation errors, and examples of adaptive execution and learned QO approaches, highlighting both opportunities and deployment challenges. The authors advocate a path toward a dynamic, robust, and adaptable QO ecosystem that can operate across converged lakehouse platforms and open-standard components, ultimately improving production performance without wholesale replumbing of core optimizers.
Abstract
For nearly half a century, the core design of query optimizers in industrial database systems has remained remarkably stable, relying on foundational principles from System R and the Volcano/Cascades framework. However, the rise of cloud computing, massive data volumes, and unified data platforms has exposed the limitations of this traditional, monolithic architecture. Taking an industrial perspective, this paper reviews the past and present of query optimization in production systems and identifies the challenges they face today. Then this paper highlights three key trends gaining momentum in the industry that promise to address these challenges. First, a tighter feedback loop between query optimization and query execution is being used to improve the robustness of query performance. Second, the scope of optimization is expanding from a single query to entire workloads through the convergence of query optimization and workload optimization. Third, and perhaps most transformatively, the industry is moving from monolithic designs to composable architectures that foster agility and cross-engine collaboration. Together, these trends chart a clear path toward a more dynamic, holistic, and adaptable future for query optimization in practice.