Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sampling in Cloud Benchmarking: A Critical Review and Methodological Guidelines

Saman Akbari, Manfred Hauswirth

TL;DR

This paper critically reviews sampling practices in cloud benchmarking by analyzing 115 recent studies across four dimensions: sampling method, sample origin, sample size, and sample availability. It finds pervasive non-probability sampling, heavy reliance on a single benchmark, and limited access to benchmarks and data, raising concerns about generalizability and reproducibility. To address this, the authors propose four methodological guidelines to improve transparency: clear sampling rationale, transparent sampling description, acknowledgment of potential sampling bias, and open access to artifacts, with examples of adherence. The study also provides an open replication package to enable replication and further meta-analysis, and it calls for community-wide development of standardized benchmarks to improve comparability.

Abstract

Cloud benchmarks suffer from performance fluctuations caused by resource contention, network latency, hardware heterogeneity, and other factors along with decisions taken in the benchmark design. In particular, the sampling strategy of benchmark designers can significantly influence benchmark results. Despite this well-known fact, no systematic approach has been devised so far to make sampling results comparable and guide benchmark designers in choosing their sampling strategy for use within benchmarks. To identify systematic problems, we critically review sampling in recent cloud computing research. Our analysis identifies concerning trends: (i) a high prevalence of non-probability sampling, (ii) over-reliance on a single benchmark, and (iii) restricted access to samples. To address these issues and increase transparency in sampling, we propose methodological guidelines for researchers and reviewers. We hope that our work contributes to improving the generalizability, reproducibility, and reliability of research results.

Sampling in Cloud Benchmarking: A Critical Review and Methodological Guidelines

TL;DR

This paper critically reviews sampling practices in cloud benchmarking by analyzing 115 recent studies across four dimensions: sampling method, sample origin, sample size, and sample availability. It finds pervasive non-probability sampling, heavy reliance on a single benchmark, and limited access to benchmarks and data, raising concerns about generalizability and reproducibility. To address this, the authors propose four methodological guidelines to improve transparency: clear sampling rationale, transparent sampling description, acknowledgment of potential sampling bias, and open access to artifacts, with examples of adherence. The study also provides an open replication package to enable replication and further meta-analysis, and it calls for community-wide development of standardized benchmarks to improve comparability.

Abstract

Cloud benchmarks suffer from performance fluctuations caused by resource contention, network latency, hardware heterogeneity, and other factors along with decisions taken in the benchmark design. In particular, the sampling strategy of benchmark designers can significantly influence benchmark results. Despite this well-known fact, no systematic approach has been devised so far to make sampling results comparable and guide benchmark designers in choosing their sampling strategy for use within benchmarks. To identify systematic problems, we critically review sampling in recent cloud computing research. Our analysis identifies concerning trends: (i) a high prevalence of non-probability sampling, (ii) over-reliance on a single benchmark, and (iii) restricted access to samples. To address these issues and increase transparency in sampling, we propose methodological guidelines for researchers and reviewers. We hope that our work contributes to improving the generalizability, reproducibility, and reliability of research results.

Paper Structure

This paper contains 24 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Sampling Methods
  4. Challenges of Sampling in Cloud Benchmarking
  5. Study Design
  6. Literature Databases
  7. Search String
  8. Selection Strategy
  9. Study Results
  10. Sampling Method
  11. Sample Origin
  12. Sample Size
  13. Sample Availability
  14. Discussion
  15. Sampling Method
  16. ...and 9 more sections

Figures (4)

  • Figure 1: Sampling methods in quantitative research.
  • Figure 2: Literature sampling in our critical review.
  • Figure 3: Empirical cumulative distribution of benchmarks ($n = 107$ studies) and executions ($n = 59$ studies). Left dashed line: p25, dotted line: median, right dashed line: p75.
  • Figure 4: Evaluation of sampling guidelines G1--G4 ($n = 115$ studies).