Lower Bounds for the Algorithmic Complexity of Learned Indexes

TL;DR

The paper provides a general, model-class–aware framework to derive lower bounds on query time for learned indexes by connecting the prediction error to CDF approximation errors and then to search costs. Using a DAG-based index representation with K sinks, it proves an $\Omega(\log(n/K))$ worst-case query-time bound under broad modeling assumptions, implying that substantial asymptotic gains over B+trees require near-linear space ($K=\Omega(n)$). It sharpens these results for piecewise-constant models via exact quantization, showing $R_{K,P_0}(F)=1/(4K)$ when data and queries share the same distribution, and extends to piecewise-linear models by proving $R_{K,P_1}(F)=\Omega(1/K)$ in a worst-case sense, using adversarial CDF constructions and Kolmogorov widths. The findings illuminate fundamental limits of learned indexes, guiding data-dependent model selection and space allocation decisions for future index designs.

Abstract

Learned index structures aim to accelerate queries by training machine learning models to approximate the rank function associated with a database attribute. While effective in practice, their theoretical limitations are not fully understood. We present a general framework for proving lower bounds on query time for learned indexes, expressed in terms of their space overhead and parameterized by the model class used for approximation. Our formulation captures a broad family of learned indexes, including most existing designs, as piecewise model-based predictors. We solve the problem of lower bounding query time in two steps: first, we use probabilistic tools to control the effect of sampling when the database attribute is drawn from a probability distribution. Then, we analyze the approximation-theoretic problem of how to optimally represent a cumulative distribution function with approximators from a given model class. Within this framework, we derive lower bounds under a range of modeling and distributional assumptions, paying particular attention to the case of piecewise linear and piecewise constant model classes, which are common in practical implementations. Our analysis shows how tools from approximation theory, such as quantization and Kolmogorov widths, can be leveraged to formalize the space-time tradeoffs inherent to learned index structures. The resulting bounds illuminate core limitations of these methods.

Figures (2)

• Figure 1: Illustration of a learned index modeled as a directed acyclic graph (DAG).
• Figure 5: Optimal approximation of the CDF $F$ (blue) via a piecewise constant function $h$ (red) with $K=5$ segments when $f=g$---i.e., when the dataset and queries are drawn from the same distribution. In this case, the optimal approximating values $\{ c_k \}$ are evenly spaced (marked on the $y$-axis), and approximating the CDF is equivalent to quantizing $\text{Uniform}([0,1])$ along the $y$-axis. The quantization boundaries (unmarked black horizontal dashed lines) are reflected onto the $x$-axis via $F^{-1}$ to split $\mathcal{Q}$ into $K=5$ equiprobable regions $\{ I_k \}$ along the quintiles of $F$ (black vertical dashed lines). The green dotted lines illustrate how the distribution of $X\sim F$ becomes uniform when transformed via $F(X)$.

Theorems & Definitions (14)

• Proposition 1
• Proposition 2
• Lemma 3: Expected bounds
• Lemma 4: Existence of small-deviations
• Proposition 5: Lower bounds for query time via prediction error
• Lemma 6
• Lemma 7: Scaling of Approximation Error
• Lemma 8: Paley–Zygmund inequality
• Lemma 9
• Lemma 10: Entropy bound under partial mass