Redbench: Workload Synthesis From Cloud Traces

TL;DR

Redbench tackles the gap between production cloud workloads and synthetic benchmarks by introducing a trace-driven benchmark that converts cloud workload traces into executable query streams on familiar schemas. It preserves core production signals—exact and scanset repetitions, temporal bursts, and interleaved reads and writes—through two complementary synthesis modes: matching-based reuse of benchmark templates and generation-based construction of novel SQL. Empirical evaluation across four commercial cloud data warehouses shows Redbench reproduces production-like characteristics and reveals caching benefits that traditional benchmarks miss, with speedups up to $2.98\times$ under realistic workloads. The work provides a practical, reproducible foundation for studying workload-driven optimizations in modern cloud data warehouses and releases its artifacts to accelerate community adoption.

Abstract

Workload traces from cloud data warehouse providers reveal that standard benchmarks such as TPC-H and TPC-DS fail to capture key characteristics of real-world workloads, including query repetition and string-heavy queries. In this paper, we introduce Redbench, a novel benchmark featuring a workload generator that reproduces real-world workload characteristics derived from traces released by cloud providers. Redbench integrates multiple workload generation techniques to tailor workloads to specific objectives, transforming existing benchmarks into realistic query streams that preserve intrinsic workload characteristics. By focusing on inherent workload signals rather than execution-specific metrics, Redbench bridges the gap between synthetic and real workloads. Our evaluation shows that (1) Redbench produces more realistic and reproducible workloads for cloud data warehouse benchmarking, and (2) Redbench reveals the impact of system optimizations across four commercial data warehouse platforms. We believe that Redbench provides a crucial foundation for advancing research on optimization techniques for modern cloud data warehouses.

Figures (9)

• Figure 1: Result caching speedups for traditional benchmarks versus production workloads. Traditional benchmarks (e.g., JOB, CEB) fail to capture workload-driven effects, such as query repetitions and temporal correlations, that are present in production traces (e.g., Redset) and are essential for evaluating optimizations like result caching.
• Figure 2: Redbench uses a given workload trace and data set as a basis for workload synthesis. Currently, Redbench supports two synthesis approaches that users can utilize to generate benchmarks with trace-shaped queries.
• Figure 3: Redbench transforms real workload traces ⓪ into executable benchmark workloads. It extracts query repetitions, temporal patterns, and access characteristics from Redset ① and reproduces them by either mapping to existing benchmark queries ② or generating new SQL ③, enabling realistic, workload-driven evaluation of DBMS optimizations ④.
• Figure 4: Redbench faithfully preserves the temporal arrival patterns from Redset, including distinct bursts of activity followed by idle periods. Both generation-based and matching-based approaches maintain exact arrival times, ensuring realistic temporal dynamics for evaluating workload-driven optimizations.
• Figure 5: Comparison of query repetition rate (QRR) and scan-set repetition rate (SRR) between Redbench and Redset. DML-aware metrics consider cache invalidations from interleaved writes on the same tables. Generation-based synthesis closely matches Redset’s repetition characteristics, while matching-based exhibits higher repetition due to the limited query pool. The baseline fails to replicate realistic repetition patterns.