Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Should I Run My Cloud Benchmark on Black Friday?

Sören Henning, Adriano Vogel, Esteban Perez-Wohlfeil, Otmar Ertl, Rick Rabiser

TL;DR

This work evaluates how cloud-based benchmarking results vary at the application level, addressing concerns about reproducibility in multi-tenant environments. It builds on prior longitudinal measurements by expanding the dataset and adding analysis around a major global event to quantify its impact. Using ShuffleBench with Kafka Streams on AWS EKS, the study finds a modest overall variability (CV ≈ 3.69%) with identifiable daily and weekly patterns and a small but detectable effect around Black Friday. The findings suggest that while cloud benchmarks are inherently variable, the observed differences are typically small (often <5%), guiding researchers and practitioners on when and how to account for variability in performance comparisons and benchmarking campaigns.

Abstract

Benchmarks and performance experiments are frequently conducted in cloud environments. However, their results are often treated with caution, as the presumed high variability of performance in the cloud raises concerns about reproducibility and credibility. In a recent study, we empirically quantified the impact of this variability on benchmarking results by repeatedly executing a stream processing application benchmark at different times of the day over several months. Our analysis confirms that performance variability is indeed observable at the application level, although it is less pronounced than often assumed. The larger scale of our study compared to related work allowed us to identify subtle daily and weekly performance patterns. We now extend this investigation by examining whether a major global event, such as Black Friday, affects the outcomes of performance benchmarks.

Should I Run My Cloud Benchmark on Black Friday?

TL;DR

This work evaluates how cloud-based benchmarking results vary at the application level, addressing concerns about reproducibility in multi-tenant environments. It builds on prior longitudinal measurements by expanding the dataset and adding analysis around a major global event to quantify its impact. Using ShuffleBench with Kafka Streams on AWS EKS, the study finds a modest overall variability (CV ≈ 3.69%) with identifiable daily and weekly patterns and a small but detectable effect around Black Friday. The findings suggest that while cloud benchmarks are inherently variable, the observed differences are typically small (often <5%), guiding researchers and practitioners on when and how to account for variability in performance comparisons and benchmarking campaigns.

Abstract

Benchmarks and performance experiments are frequently conducted in cloud environments. However, their results are often treated with caution, as the presumed high variability of performance in the cloud raises concerns about reproducibility and credibility. In a recent study, we empirically quantified the impact of this variability on benchmarking results by repeatedly executing a stream processing application benchmark at different times of the day over several months. Our analysis confirms that performance variability is indeed observable at the application level, although it is less pronounced than often assumed. The larger scale of our study compared to related work allowed us to identify subtle daily and weekly performance patterns. We now extend this investigation by examining whether a major global event, such as Black Friday, affects the outcomes of performance benchmarks.
Paper Structure (15 sections, 4 figures)

This paper contains 15 sections, 4 figures.

Table of Contents

  1. Introduction
  2. Experiment Design
  3. Test subject.
  4. Performance measurements.
  5. Execution environment.
  6. Automated benchmark execution.
  7. Time spans.
  8. Cloud Performance Patterns
  9. Daily pattern.
  10. Weekly pattern.
  11. Long-term pattern.
  12. Further observations.
  13. Cloud Performance on Black Friday
  14. Conclusions
  15. Acknowledgments

Figures (4)

  • Figure 1: Measured throughput summarized by the hour of the day; color intensity reflects sample size FSE2025.
  • Figure 2: Measured throughput summarized by the day of the week; color intensity reflects sample size FSE2025.
  • Figure 3: Measured throughput in benchmark executions around Black Friday 2024.
  • Figure 4: Mean daily throughput around Black Friday compared to the daily pattern described in \ref{['sec:patterns']}.