When Should I Run My Application Benchmark?: Studying Cloud Performance Variability for the Case of Stream Processing Applications

TL;DR

The paper tackles how cloud performance variability affects end-to-end application benchmarks, focusing on distributed stream processing. It combines benchmarking and case-study methods to analyze ShuffleBench running on AWS EKS with Kafka Streams over more than three months, quantifying variability and temporal patterns. Key findings show the variability is real but modest ($CV$ around $3.69\%$), with subtle daily and weekly patterns and a small benefit from re-provisioning benchmarking infrastructure. The work offers practical guidelines for when to run benchmarks, highlights region and instance-type effects, and provides an open replication package to facilitate future cloud-variability studies.

Abstract

Performance benchmarking is a common practice in software engineering, particularly when building large-scale, distributed, and data-intensive systems. While cloud environments offer several advantages for running benchmarks, it is often reported that benchmark results can vary significantly between repetitions -- making it difficult to draw reliable conclusions about real-world performance. In this paper, we empirically quantify the impact of cloud performance variability on benchmarking results, focusing on stream processing applications as a representative type of data-intensive, performance-critical system. In a longitudinal study spanning more than three months, we repeatedly executed an application benchmark used in research and development at Dynatrace. This allows us to assess various aspects of performance variability, particularly concerning temporal effects. With approximately 591 hours of experiments, deploying 789 Kubernetes clusters on AWS and executing 2366 benchmarks, this is likely the largest study of its kind and the only one addressing performance from an end-to-end, i.e., application benchmark perspective. Our study confirms that performance variability exists, but it is less pronounced than often assumed (coefficient of variation of < 3.7%). Unlike related studies, we find that performance does exhibit a daily and weekly pattern, although with only small variability (<= 2.5%). Re-using benchmarking infrastructure across multiple repetitions introduces only a slight reduction in result accuracy (<= 2.5 percentage points). These key observations hold consistently across different cloud regions and machine types with different processor architectures. We conclude that for engineers and researchers focused on detecting substantial performance differences (e.g., > 5%) in...

Figures (9)

• Figure 1: Histogram and associated boxplot of all throughput measurements in our baseline experiments.
• Figure 2: Measured throughput summarized by the hour of the day. The color intensity indicates the sample size.
• Figure 3: Measured throughput summarized by the day of the week. The color intensity indicates the sample size.
• Figure 4: Measured throughput summarized by the ISO week number with zoomed-in y-axis range. The color intensity indicates the sample size.
• Figure 5: ECDF of difference between minimum and maximum of three observed value when re-using the benchmark infrastructure and when re-creating it before each execution.