CardOOD: Robust Query-driven Cardinality Estimation under Out-of-Distribution

TL;DR

CardOOD tackles out-of-distribution generalization in query-driven cardinality estimation by offering an offline-training framework that can wrap existing DL-based estimators. It combines three robust-learning categories—representation learning, data manipulation, and new learning strategies—and introduces OrderEmb, a self-supervised objective modeling partial order among queries. Across datasets such as forest, IMDB, DSB, and JOB-light, CardOOD-based estimators reduce tail q-errors and, when integrated into PostgreSQL, yield practical end-to-end speedups (up to $5.6\%$, and $26\%$–$36\%$ in various workloads). The approach is model-agnostic, complementing existing transfer/robust learning techniques and enabling robust initialization and periodic retraining for real-world DBMS optimization.

Abstract

Query-driven learned estimators are accurate, flexible, and lightweight alternatives to traditional estimators in query optimization. However, existing query-driven approaches struggle with the Out-of-distribution (OOD) problem, where the test workload distribution differs from the training workload, leading to performancedegradation. In this paper, we present CardOOD, a general learning framework designed to construct robust query-driven cardinality estimators that are resilient against the OOD problem. Our framework focuses on offline training algorithms that develop one-off models from a static workload, suitable for model initialization and periodic retraining. In CardOOD, we extend classical transfer/robust learning techniques to train query-driven cardinalityestimators, and the algorithms fall into three categories: representation learning, data manipulation, and new learning strategies. As these learning techniques are originally evaluated in computervision tasks, we also propose a new learning algorithm that exploits the property of cardinality estimation. This algorithm, lying in the category of new learning strategy, models the partial order constraint of cardinalities by a self-supervised learning task. Comprehensive experimental studies demonstrate the efficacy of the algorithms of CardOOD in mitigating the OOD problem to varying extents. We further integrate CardOOD into PostgreSQL, showcasing its practical utility in query optimization.

Figures (5)

• Figure 1: Varying # of Training Queries
• Figure 2: Training Time (Seconds)
• Figure 3: Varying # of Contrastive Queries in OrderEmbed
• Figure 4: Query Speedup in PostgreSQL for $$IMDB: # Table in the Test Queries is OOD
• Figure 5: Query Speedup in PostgreSQL for $$DSB and $$JOB-$$light: Test Query Template is OOD