Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Information Theory Strikes Back: New Development in the Theory of Cardinality Estimation

Mahmoud Abo Khamis, Vasileios Nakos, Dan Olteanu, Dan Suciu

TL;DR

The paper reframes cardinality estimation for conjunctive queries as an information-theoretic optimization problem by using $\ell_p$-norms of degree sequences to derive upper bounds on query outputs. Bounds are computed as the optimal value of a linear program that maximizes joint entropy under Shannon inequalities and $\ell_p$-norm constraints, generalizing AGM and PANDA to arbitrary norms. The approach yields a polymatroid-based dual characterization (Log-L-Bound) and a practical evaluation algorithm that reduces to PANDA via careful partitioning. Preliminary experiments on Berge-acyclic and triangle-join queries show that $\ell_p$-norm bounds can closely track true cardinalities and outperform traditional and some learned estimators, though they may overestimate when inputs are highly miscalibrated.

Abstract

Estimating the cardinality of the output of a query is a fundamental problem in database query processing. In this article, we overview a recently published contribution that casts the cardinality estimation problem as linear optimization and computes guaranteed upper bounds on the cardinality of the output for any full conjunctive query. The objective of the linear program is to maximize the joint entropy of the query variables and its constraints are the Shannon information inequalities and new information inequalities involving $\ell_p$-norms of the degree sequences of the join attributes. The bounds based on arbitrary norms can be asymptotically lower than those based on the $\ell_1$ and $\ell_\infty$ norms, which capture the cardinalities and respectively the max-degrees of the input relations. They come with a matching query evaluation algorithm, are computable in exponential time in the query size, and are provably tight when each degree sequence is on one join attribute.

Information Theory Strikes Back: New Development in the Theory of Cardinality Estimation

TL;DR

The paper reframes cardinality estimation for conjunctive queries as an information-theoretic optimization problem by using -norms of degree sequences to derive upper bounds on query outputs. Bounds are computed as the optimal value of a linear program that maximizes joint entropy under Shannon inequalities and -norm constraints, generalizing AGM and PANDA to arbitrary norms. The approach yields a polymatroid-based dual characterization (Log-L-Bound) and a practical evaluation algorithm that reduces to PANDA via careful partitioning. Preliminary experiments on Berge-acyclic and triangle-join queries show that -norm bounds can closely track true cardinalities and outperform traditional and some learned estimators, though they may overestimate when inputs are highly miscalibrated.

Abstract

Estimating the cardinality of the output of a query is a fundamental problem in database query processing. In this article, we overview a recently published contribution that casts the cardinality estimation problem as linear optimization and computes guaranteed upper bounds on the cardinality of the output for any full conjunctive query. The objective of the linear program is to maximize the joint entropy of the query variables and its constraints are the Shannon information inequalities and new information inequalities involving -norms of the degree sequences of the join attributes. The bounds based on arbitrary norms can be asymptotically lower than those based on the and norms, which capture the cardinalities and respectively the max-degrees of the input relations. They come with a matching query evaluation algorithm, are computable in exponential time in the query size, and are provably tight when each degree sequence is on one join attribute.

Paper Structure

This paper contains 12 sections, 5 theorems, 45 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. The $\ell_p$-Bounds in Action
  3. Preliminaries
  4. Queries and Data Statistics
  5. Background in Information Theory
  6. Bounds on Query Output Size
  7. Demystifying the L$_p$-norm Information Inequality
  8. Algorithm Meeting the Bound
  9. Preliminary Experiments
  10. Experiments with Berge-Acyclic Queries
  11. Experiments with Triangle Join Queries
  12. Conclusions

Key Result

Theorem 3.1

Let $Q$ be a query eq:full:cq, $\bm U_i, \bm V_i\subseteq \bm X$ be sets of variables, for $i\in[s]$, and suppose that the following information inequality is valid for all entropic vectors $\bm h$ with variables $\bm X$: where $w_i \geq 0$, and $p_i \in (0,\infty]$, for all $i\in[s]$. Assume that each conditional $(\bm V_i|\bm U_i)$ in eq:ii:lp is guarded by some relation $R_{j_i}$ in $Q$. Then,

Figures (3)

  • Figure 1: Relation $R$ used in Example \ref{['ex:degree-sequence']}. The degree sequence $\texttt{deg}_R(Y|X)$ states how many $Y$-values are paired with each $X$-value in $R$. The degree sequence $\texttt{deg}_R(X|Y)$ states how many $X$-values are paired with each $Y$-value in $R$.
  • Figure 2: Relations $(R_p)_{p\geq 1}$ defined as the union of $k$ Cartesian products of blocks of values for the attributes $A$, $B_1,\ldots,B_p$.
  • Figure 3: Estimation errors for 31 Berge-acyclic full conjunctive queries in the JOBjoin benchmark.

Theorems & Definitions (12)

  • Example 1.1
  • Example 1.2
  • Example 2.1
  • Theorem 3.1: LpBound:PODS:2024
  • Example 3.2
  • Example 3.3
  • Theorem 3.4
  • Example 3.5
  • Example 3.6
  • Lemma 5.1
  • ...and 2 more