PDSP-Bench: A Benchmarking System for Parallel and Distributed Stream Processing

TL;DR

PDSP-Bench introduces a benchmarking system for parallel and distributed stream processing in heterogeneous environments, enabling large-scale, diverse workloads and ML-informed evaluation. It features a workload generator, a controller, and a web UI to create realistic PQPs across real-world and synthetic apps, deploy them on diverse hardware, and train/compare learned cost estimators. Through extensive experiments with Apache Flink, the work reveals non-linear effects of parallelism, hardware diversity trade-offs, and data-efficient training strategies for graph-based latency predictors. The system offers a scalable, extensible platform for fair comparison of SPS workloads and optimization approaches, with practical implications for deploying and tuning real-time streaming pipelines.

Abstract

The paper introduces PDSP-Bench, a novel benchmarking system designed for a systematic understanding of performance of parallel stream processing in a distributed environment. Such an understanding is essential for determining how Stream Processing Systems (SPS) use operator parallelism and the available resources to process massive workloads of modern applications. Existing benchmarking systems focus on analyzing SPS using queries with sequential operator pipelines within a homogeneous centralized environment. Quite differently, PDSP-Bench emphasizes the aspects of parallel stream processing in a distributed heterogeneous environment and simultaneously allows the integration of machine learning models for SPS workloads. In our results, we benchmark a well-known SPS, Apache Flink, using parallel query structures derived from real-world applications and synthetic queries to show the capabilities of PDSP-Bench towards parallel stream processing. Moreover, we compare different learned cost models using generated SPS workloads on PDSP-Bench by showcasing their evaluations on model and training efficiency. We present key observations from our experiments using PDSP-Bench that highlight interesting trends given different query workloads, such as non-linearity and paradoxical effects of parallelism on the performance.

Figures (6)

• Figure 1: PDSP-Bench system overview
• Figure 2: Example of pqp: synthetic 2-way join and real-world ad analytics application.
• Figure 3: Impact of parallelism degree on pqp performance from synthetic (top) and real-world applications (bottom). The analysis shows distinct performance behaviors, highlighting the benefits of parallelism in improving end-to-end latency and the need to consider query and operator characteristics in PDSP environments. Results indicate a complex interplay between standard operations and udo, with paradoxical effects and non-linear performance trends related to parallelism degree. (Note: Bottom figure omits XS and XL; their performance mirrors S and L)
• Figure 4: Impact of heterogeneous hardware on performance for pdsp, with varying parallelism degree and resource processing capabilities on real-world (top) and synthetic (bottom) applications. Evaluation shows that parallel processing benefits from hardware diversity but requires understanding hardware characteristics and workload distribution to optimize performance and avoid of pitfalls such as diversity dilemma.
• Figure 5: Comparison of accuracy of various ML models: Linear regression (LR) ganapathiFlatVector2009, Multi-layer perceptron (MLP) hosseinzadeh2014multilayer, Random forest (RF) chen2016parallel, Graph neural networks (GNN) wu2024stagePrAg_ZeroTune_ICDE_2024heinrich2024costream for various parallel query structures.