Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Unified Model for Cardinality Estimation by Learning from Data and Queries via Sum-Product Networks

Jiawei Liu, Ju Fan, Tongyu Liu, Kai Zeng, Jiannan Wang, Quehuan Liu, Tao Ye, Nan Tang

TL;DR

Cardinality estimation must balance accuracy, speed, and storage in realistic DBMSs. The paper presents QSPN, a query-aware Sum-Product Network that jointly learns from data and query workloads by introducing QProduct and QSplit to partition columns and workloads. It provides offline construction and online inference algorithms, plus an extension to multi-table cardinality estimation via join-key binning. Empirical results show QSPN delivers superior accuracy, faster inference, and competitive storage across single- and multi-table benchmarks and end-to-end queries. The approach offers a practical, scalable path toward robust CardEst in real-world analytics workloads.

Abstract

Cardinality estimation is a fundamental component in database systems, crucial for generating efficient execution plans. Despite advancements in learning-based cardinality estimation, existing methods may struggle to simultaneously optimize the key criteria: estimation accuracy, inference time, and storage overhead, limiting their practical applicability in real-world database environments. This paper introduces QSPN, a unified model that integrates both data distribution and query workload. QSPN achieves high estimation accuracy by modeling data distribution using the simple yet effective Sum-Product Network (SPN) structure. To ensure low inference time and reduce storage overhead, QSPN further partitions columns based on query access patterns. We formalize QSPN as a tree-based structure that extends SPNs by introducing two new node types: QProduct and QSplit. This paper studies the research challenges of developing efficient algorithms for the offline construction and online computation of QSPN. We conduct extensive experiments to evaluate QSPN in both single-table and multi-table cardinality estimation settings. The experimental results have demonstrated that QSPN achieves superior and robust performance on the three key criteria, compared with state-of-the-art approaches.

A Unified Model for Cardinality Estimation by Learning from Data and Queries via Sum-Product Networks

TL;DR

Cardinality estimation must balance accuracy, speed, and storage in realistic DBMSs. The paper presents QSPN, a query-aware Sum-Product Network that jointly learns from data and query workloads by introducing QProduct and QSplit to partition columns and workloads. It provides offline construction and online inference algorithms, plus an extension to multi-table cardinality estimation via join-key binning. Empirical results show QSPN delivers superior accuracy, faster inference, and competitive storage across single- and multi-table benchmarks and end-to-end queries. The approach offers a practical, scalable path toward robust CardEst in real-world analytics workloads.

Abstract

Cardinality estimation is a fundamental component in database systems, crucial for generating efficient execution plans. Despite advancements in learning-based cardinality estimation, existing methods may struggle to simultaneously optimize the key criteria: estimation accuracy, inference time, and storage overhead, limiting their practical applicability in real-world database environments. This paper introduces QSPN, a unified model that integrates both data distribution and query workload. QSPN achieves high estimation accuracy by modeling data distribution using the simple yet effective Sum-Product Network (SPN) structure. To ensure low inference time and reduce storage overhead, QSPN further partitions columns based on query access patterns. We formalize QSPN as a tree-based structure that extends SPNs by introducing two new node types: QProduct and QSplit. This paper studies the research challenges of developing efficient algorithms for the offline construction and online computation of QSPN. We conduct extensive experiments to evaluate QSPN in both single-table and multi-table cardinality estimation settings. The experimental results have demonstrated that QSPN achieves superior and robust performance on the three key criteria, compared with state-of-the-art approaches.
Paper Structure (34 sections, 6 theorems, 7 equations, 8 figures, 4 tables, 7 algorithms)

This paper contains 34 sections, 6 theorems, 7 equations, 8 figures, 4 tables, 7 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Problem Formalization
  4. SUM-Product Network (SPN) Model
  5. An Overview of QSPN
  6. Offline QSPN Construction
  7. Construction of QProduct
  8. Construction of QSplit
  9. Online QSPN Computation
  10. CardEst Inference with QSPN
  11. QSPN Model Update
  12. Multi-Table CardEst with QSPN
  13. Experiments
  14. Experimental Setting
  15. Evaluation on Single-Table CardEst
  16. ...and 19 more sections

Key Result

Lemma 1

The problem of QProduct construction is equivalent to the minimum $k$-cut problem.

Figures (8)

  • Figure 1: High-level idea of QSPN.
  • Figure 2: An example of QSplit construction.
  • Figure 3: Illustration of CardEst Inference with QSPN.
  • Figure 4: Illustration of our proposed M-QSPN method.
  • Figure 5: Evaluation on Dynamic Model Update (Hybrid-Update Setting).
  • ...and 3 more figures

Theorems & Definitions (13)

  • Example 1
  • definition 1: Query-Column Access Matrix
  • definition 2: Access Affinity
  • definition 3: QProduct
  • Lemma 1
  • Example 2
  • definition 4: QSplit
  • definition 5: Upper Bound
  • Lemma 2
  • Lemma 3
  • ...and 3 more