Stochastic SketchRefine: Scaling In-Database Decision-Making under Uncertainty to Millions of Tuples

TL;DR

The paper tackles scaling in-database stochastic optimization for SPQs by introducing risk-constrained linearization (RCL) to convert VaR/CVaR constraints into tractable linear forms, and a two-phase Stochastic SketchRefine framework that partitions large relations and solves over inexpensive sketches before refining with real tuples. The combination yields near-optimal packages with dramatically reduced runtimes, enabling handling of datasets with millions of tuples and high-variance stochastic attributes. Key contributions include extending SPaQL with CVaR constraints, the RCL-Solve solver with alternating parameter search, and DistPartition-based stochastic partitioning plus a robust sketch-refine pipeline with NORTA-based correlation modeling. The practical impact is substantial: enabling scalable, uncertainty-aware decision-making directly in-database for very large data volumes, with strong theoretical guarantees and demonstrated empirical scalability and quality.

Abstract

Decision making under uncertainty often requires choosing packages, or bags of tuples, that collectively optimize expected outcomes while limiting risks. Processing Stochastic Package Queries (SPQs) involves solving very large optimization problems on uncertain data. Monte Carlo methods create numerous scenarios, or sample realizations of the stochastic attributes of all the tuples, and generate packages with optimal objective values across these scenarios. The number of scenarios needed for accurate approximation - and hence the size of the optimization problem when using prior methods - increases with variance in the data, and the search space of the optimization problem increases exponentially with the number of tuples in the relation. Existing solvers take hours to process SPQs on large relations containing stochastic attributes with high variance. Besides enriching the SPaQL language to capture a broader class of risk specifications, we make two fundamental contributions towards scalable SPQ processing. First, to handle high variance, we propose risk-constraint linearization (RCL), which converts SPQs into Integer Linear Programs (ILPs) whose size is independent of the number of scenarios used. Solving these ILPs gives us feasible and near-optimal packages. Second, we propose Stochastic SketchRefine, a divide and conquer framework that breaks down a large stochastic optimization problem into subproblems involving smaller subsets of tuples. Our experiments show that, together, RCL and Stochastic SketchRefine produce high-quality packages in orders of magnitude lower runtime than the state of the art.

Figures (9)

• Figure 1: The gain in the Stock_Investments table is an uncertain attribute, simulated by stochastic processes. The scenarios represent different simulations (possible worlds). The example SPaQL query contains a value-at-risk (VaR) constraint, specifying that the probability of total loss (negative gain) exceeding $10 is at most 5%.
• Figure 2: Solving a stochastic package query (SPQ) with Stochastic SketchRefine.
• Figure 3: Increasing variance or volatility coefficients increases tuple uncertainty as well as query hardness. ($H$ range reported with each plot). RCL-Solve is faster than SummarySearch especially at high variances. In each plot, $\mu$ and $\sigma$ report the mean and standard deviation, respectively, of the relative integrality gap for RCL-Solve's packages.
• Figure 4: RCL-Solve continues to outperform SummarySearch but fails to scale beyond 1M tuples. The absence of data points indicates that no solutions were found within 1.5 hours. Stochastic SketchRefine scales well as data size and query hardness ($H$) increase. Each plot shows the relative integrality gap statistics ($\mu, \sigma$) for Stochastic SketchRefine's packages.
• Figure 5: MAD for correlated Pareto$(a,\alpha)$ and uniform$[0,100]$ random variables for different values of NORTA correlation coefficient $\rho$, Pareto mean $\mu=\bigl(\alpha/(\alpha-1)\bigr)a$, and Pareto tail coefficient $\alpha$.

Theorems & Definitions (5)

• theorem 1
• theorem 2
• theorem 3
• theorem 4: Approximation Guarantee
• theorem 5