Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NeurBench: Benchmarking Learned Database Components with Data and Workload Drift Modeling

Zhanhao Zhao, Haotian Gao, Naili Xing, Lingze Zeng, Meihui Zhang, Gang Chen, Manuel Rigger, Beng Chin Ooi

TL;DR

NeurBench tackles the challenge of evaluating learned database components under data and workload drift by introducing a drift factor that quantifies drift via distribution distance. It combines a drift-aware diffusion-based data/workload generator (Diffuser+Drifter) with a diffusion-guided framework to synthesize drifted data while preserving correlations, enabling controlled, realistic drift scenarios. The paper demonstrates NeurBench’s ability to generate drifted datasets and workloads and uses it to benchmark state-of-the-art learned query optimizers, learned indexes, and learned concurrency control, revealing design trade-offs and robustness under drift. This work provides a practical pathway to assess adaptability of learned DBMS components, guiding future improvements toward drift-resilient designs and adaptive strategies.

Abstract

Learned database components, which deeply integrate machine learning into their design, have been extensively studied in recent years. Given the dynamism of databases, where data and workloads continuously drift, it is crucial for learned database components to remain effective and efficient in the face of data and workload drift. Adaptability, therefore, is a key factor in assessing their practical applicability. However, existing benchmarks for learned database components either overlook or oversimplify the treatment of data and workload drift, failing to evaluate learned database components across a broad range of drift scenarios. This paper presents NeurBench, a new benchmark suite that applies measurable and controllable data and workload drift to enable systematic performance evaluations of learned database components. We quantify diverse types of drift by introducing a key concept called the drift factor. Building on this formulation, we propose a drift-aware data and workload generation framework that effectively simulates real-world drift while preserving inherent correlations. We employ NeurBench to evaluate state-of-the-art learned query optimizers, learned indexes, and learned concurrency control within a consistent experimental process, providing insights into their performance under diverse data and workload drift scenarios.

NeurBench: Benchmarking Learned Database Components with Data and Workload Drift Modeling

TL;DR

NeurBench tackles the challenge of evaluating learned database components under data and workload drift by introducing a drift factor that quantifies drift via distribution distance. It combines a drift-aware diffusion-based data/workload generator (Diffuser+Drifter) with a diffusion-guided framework to synthesize drifted data while preserving correlations, enabling controlled, realistic drift scenarios. The paper demonstrates NeurBench’s ability to generate drifted datasets and workloads and uses it to benchmark state-of-the-art learned query optimizers, learned indexes, and learned concurrency control, revealing design trade-offs and robustness under drift. This work provides a practical pathway to assess adaptability of learned DBMS components, guiding future improvements toward drift-resilient designs and adaptive strategies.

Abstract

Learned database components, which deeply integrate machine learning into their design, have been extensively studied in recent years. Given the dynamism of databases, where data and workloads continuously drift, it is crucial for learned database components to remain effective and efficient in the face of data and workload drift. Adaptability, therefore, is a key factor in assessing their practical applicability. However, existing benchmarks for learned database components either overlook or oversimplify the treatment of data and workload drift, failing to evaluate learned database components across a broad range of drift scenarios. This paper presents NeurBench, a new benchmark suite that applies measurable and controllable data and workload drift to enable systematic performance evaluations of learned database components. We quantify diverse types of drift by introducing a key concept called the drift factor. Building on this formulation, we propose a drift-aware data and workload generation framework that effectively simulates real-world drift while preserving inherent correlations. We employ NeurBench to evaluate state-of-the-art learned query optimizers, learned indexes, and learned concurrency control within a consistent experimental process, providing insights into their performance under diverse data and workload drift scenarios.

Paper Structure

This paper contains 57 sections, 1 theorem, 14 equations, 13 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Learned Database Components
  4. Data and Workload Drift
  5. Denoising Diffusion Probabilistic Model
  6. Diffusion (Forward) Process
  7. Denoising (Reverse) Process
  8. Guided Diffusion
  9. Overview of $\textsf{NeurBench}$
  10. Drift-aware Data and Workload Generator
  11. Performance Evaluator
  12. Running Example
  13. Controlled Drift Generation
  14. Drift Factor Modeling
  15. Drift-aware Data and Workload Generation
  16. ...and 42 more sections

Key Result

Theorem 1

The controlled reverse distribution $p(\mathbf{x}_{t-1} | \mathbf{x}_t, \mathbf{x}_{drift})$ is able to be approximated using $\mathrm{Pr}(\mathbf{x}_{drift} | \mathbf{x}_t)$ and its gradient at $\mathbf{x}_t$, or $\nabla_{\mathbf{x}_t} \mathrm{Pr}(\mathbf{x}_{drift} | \mathbf{x}_t)$.

Figures (13)

  • Figure 1: System Overview of $\textsf{NeurBench}$
  • Figure 2: Diffuser and Drifter Training
  • Figure 3: End-to-end Learned Query Optimizers
  • Figure 4: Structures of Existing Learned Indexes
  • Figure 5: Performance on Generating Drifted Datasets with Varying Drift Factors
  • ...and 8 more figures

Theorems & Definitions (5)

  • Definition 1: Learned Database Component
  • Definition 2: Performance Objective
  • Definition 3: Data and Workload Drift
  • Definition 4: Drift-aware Data and Workload Generation
  • Theorem 1