Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Theoretical Framework for Distribution-Aware Dataset Search

Aryan Esmailpour, Sainyam Galhotra, Rahul Raychaudhury, Stavros Sintos

TL;DR

This work introduces a theoretical framework for distribution-aware dataset search, unifying percentile- and top-k preference-based queries under centralized and federated settings. It shows fundamental lower bounds for exact structures and then develops approximate, near-linear space data structures with polylog query time and output-sensitive performance for Pt ile and Pref, using coresets (ε-samples) and ε-nets, plus dynamic range-tree indexing with delay guarantees. The methods extend to conjunctions/disjunctions of predicates, support dynamic updates, and adapt to general synopses with known or unknown error δ, providing strong guarantees where prior heuristic approaches lacked absence of misses. The framework and data structures have practical impact for data marketplaces and data discovery tasks needing distributional guarantees despite limited data access. Overall, the paper advances distribution-aware dataset search by delivering rigorous, scalable algorithms with provable accuracy and performance in both centralized and federated contexts.

Abstract

Effective data discovery is a cornerstone of modern data-driven decision-making. Yet, identifying datasets with specific distributional characteristics, such as percentiles or preferences, remains challenging. While recent proposals have enabled users to search based on percentile predicates, much of the research in data discovery relies on heuristics. This paper presents the first theoretically backed framework that unifies data discovery under centralized and decentralized settings. Let $\mathcal{P}=\{P_1,...,P_N\}$ be a repository of $N$ datasets, where $P_i\subset \mathbb{R}^d$, for $d=O(1)$ . We study the percentile indexing (Ptile) problem and the preference indexing (Pref) problem under the centralized and the federated setting. In the centralized setting we assume direct access to the datasets. In the federated setting we assume access to a synopsis of each dataset. The goal of Ptile is to construct a data structure such that given a predicate (rectangle $R$ and interval $θ$) report all indexes $J$ such that $j\in J$ iff $|P_j\cap R|/|P_j|\inθ$. The goal of Pref is to construct a data structure such that given a predicate (vector $v$ and interval $θ$) report all indexes $J$ such that $j\in J$ iff $ω(P_j,v)\in θ$, where $ω(P_j,v)$ is the inner-product of the $k$-th largest projection of $P_j$ on $v$. We first show that we cannot hope for near-linear data structures with polylogarithmic query time in the centralized setting. Next we show $\tilde{O}(N)$ space data structures that answer Ptile and Pref queries in $\tilde{O}(1+OUT)$ time, where $OUT$ is the output size. Each data structure returns a set of indexes $J$ such that i) for every $P_i$ that satisfies the predicate, $i\in J$ and ii) if $j\in J$ then $P_j$ satisfies the predicate up to an additive error $\varepsilon+2δ$, where $\varepsilon\in(0,1)$ and $δ$ is the error of synopses.

A Theoretical Framework for Distribution-Aware Dataset Search

TL;DR

This work introduces a theoretical framework for distribution-aware dataset search, unifying percentile- and top-k preference-based queries under centralized and federated settings. It shows fundamental lower bounds for exact structures and then develops approximate, near-linear space data structures with polylog query time and output-sensitive performance for Pt ile and Pref, using coresets (ε-samples) and ε-nets, plus dynamic range-tree indexing with delay guarantees. The methods extend to conjunctions/disjunctions of predicates, support dynamic updates, and adapt to general synopses with known or unknown error δ, providing strong guarantees where prior heuristic approaches lacked absence of misses. The framework and data structures have practical impact for data marketplaces and data discovery tasks needing distributional guarantees despite limited data access. Overall, the paper advances distribution-aware dataset search by delivering rigorous, scalable algorithms with provable accuracy and performance in both centralized and federated contexts.

Abstract

Effective data discovery is a cornerstone of modern data-driven decision-making. Yet, identifying datasets with specific distributional characteristics, such as percentiles or preferences, remains challenging. While recent proposals have enabled users to search based on percentile predicates, much of the research in data discovery relies on heuristics. This paper presents the first theoretically backed framework that unifies data discovery under centralized and decentralized settings. Let be a repository of datasets, where , for . We study the percentile indexing (Ptile) problem and the preference indexing (Pref) problem under the centralized and the federated setting. In the centralized setting we assume direct access to the datasets. In the federated setting we assume access to a synopsis of each dataset. The goal of Ptile is to construct a data structure such that given a predicate (rectangle and interval ) report all indexes such that iff . The goal of Pref is to construct a data structure such that given a predicate (vector and interval ) report all indexes such that iff , where is the inner-product of the -th largest projection of on . We first show that we cannot hope for near-linear data structures with polylogarithmic query time in the centralized setting. Next we show space data structures that answer Ptile and Pref queries in time, where is the output size. Each data structure returns a set of indexes such that i) for every that satisfies the predicate, and ii) if then satisfies the predicate up to an additive error , where and is the error of synopses.

Paper Structure

This paper contains 83 sections, 31 theorems, 8 equations, 5 figures, 6 algorithms.

Table of Contents

  1. Introduction
  2. Theoretical Framework
  3. Dataset
  4. Repository
  5. Measure Function
  6. Predicates
  7. Synopsis
  8. Distribution-Aware Indexing
  9. Problem definition
  10. Percentile-aware indexing
  11. Preference-aware Indexing
  12. Our Results
  13. Related work
  14. Preliminaries
  15. Hyper-rectangles
  16. ...and 68 more sections

Key Result

lemma 1

Let $\mathcal{S}_X$ be the synopsis of $X$ with respect to the class $\mathcal{F}^d_{\square}$. Assume that $\textsc{Err}_{\mathcal{S}_X}(\mathcal{F}_{\square}^d)=\delta$ for a parameter $\delta\in[0,1)$. If $\mathsf{S}$ is a set of $O(\varepsilon^{-2}\log \varphi^{-1})$ random samples from $\mathca

Figures (5)

  • Figure 1: Toy example.
  • Figure 2: Let $\varepsilon=0.01$, $\delta=0$, and $R$ is the query rectangle. Let $w_{q_{\rho_1}}=\frac{|\mathsf{S}_j\cap \rho_1|}{|\mathsf{S}|}=0.1$ and $w_{q_{\rho_2}}=\frac{|\mathsf{S}_j\cap \rho_2|}{|\mathsf{S}|}=0.9$, while $\theta=[0, 0.2]$. If the query procedure first finds $q_{\rho_1}$ it will report the index $j$. However, the actual measure function of $P_j$ is at least $0.9-0.01\gg 0.2$. Hence $j$ should not be reported.
  • Figure 3: If $R$ is the query rectangle and $\rho$ is the largest rectangle in $R$ then $\rho\subseteq R\subset\!\!\subset\hat{\rho}$. On the other hand, for any other rectangle $\rho_1\in \mathcal{R}_j$ with $\rho_1\subset\rho$ there is no point $q_{(\rho_1,\hat{\rho}_1)}\in Q$ such that $R\subset\!\!\subset \hat{\rho}_1$ because there exists $\rho$ with $\rho_1\subset \rho\subseteq R\subset\!\!\subset\hat{\rho}_1$.
  • Figure 4: An example of the lower bound construction. The rectangle $\rho_{i, j}$ only contains the points from $G_i$ and $G'_j$. The points from same the same point sets are shown with the same color. Since red appears twice in $\rho_{i, j}$, the sets $S_i$ and $S_j$ intersect.
  • Figure : $\mathsf{ConstructTop}(\mathcal{S}_{P_1}, \ldots, \mathcal{S}_{P_N},\varepsilon)$

Theorems & Definitions (32)

  • lemma 1
  • definition 1
  • lemma 2
  • theorem 1
  • theorem 2
  • lemma 3
  • lemma 4
  • lemma 5
  • theorem 3
  • lemma 6
  • ...and 22 more