$μ$OpTime: Statically Reducing the Execution Time of Microbenchmark Suites Using Stability Metrics

TL;DR

µOpTime tackles the practicality gap of microbenchmarking in CI/CD by statically computing a per-microbenchmark minimal repetition configuration from a full-suite run, balancing accuracy and execution time with stability metrics. Operating offline, it simulates all smaller configurations and selects the shortest one that preserves result quality, yielding substantial time savings (Go up to 95.83%; Java datasets up to 71.74%) while maintaining the ability to detect regressions. The approach relies on RMIT-style multi-level repetition, stability metrics (CV, RMAD, RCIW variants), and bootstrap-based confidence intervals to quantify stability. Empirical evaluation across 14 projects in Go and Java demonstrates both strong time reductions and generally reliable regression detection, with caveats related to JVM variability and dataset-specific metric behavior, underscoring practical integration in CI/CD pipelines with careful metric selection.

Abstract

Performance regressions have a tremendous impact on the quality of software. One way to catch regressions before they reach production is executing performance tests before deployment, e.g., using microbenchmarks, which measure performance at subroutine level. In projects with many microbenchmarks, this may take several hours due to repeated execution to get accurate results, disqualifying them from frequent use in CI/CD pipelines. We propose $μ$OpTime, a static approach to reduce the execution time of microbenchmark suites by configuring the number of repetitions for each microbenchmark. Based on the results of a full, previous microbenchmark suite run, $μ$OpTime determines the minimal number of (measurement) repetitions with statistical stability metrics that still lead to accurate results. We evaluate $μ$OpTime with an experimental study on 14 open-source projects written in two programming languages and five stability metrics. Our results show that (i) $μ$OpTime reduces the total suite execution time (measurement phase) by up to 95.83% (Go) and 94.17% (Java), (ii) the choice of stability metric depends on the project and programming language, (iii) microbenchmark warmup phases have to be considered for Java projects (potentially leading to higher reductions), and (iv) $μ$OpTime can be used to reliably detect performance regressions in CI/CD pipelines.

Figures (4)

• Figure 1: RMIT schema: Each microbenchmark is executed twice within each suite run, with each iteration shuffled to a random position inside the suite run. The full suite is run twice on two cloud instances concurrently. Not shown is the benchmark duration, which determines how long each iteration is executed.
• Figure 2: µOpTime in CI/CD pipelines: The optimization runs after every major change to identify minimal execution configurations for each microbenchmark individually.
• Figure 3: Time savings of µOpTime: The x-axis shows every project from data set 1, 2, and 3. The left y-axes of all three facets show the execution time in hours of the bars. Since the projects in each data set have a different number of microbenchmarks as well as a different execution configuration, the overall execution time varies strongly across data sets. Therefore, the left y-axes are scaled differently. The right y-axis shows different fractions of microbenchmarks (MBs) with low change rates after using µOpTime, represented by marks in the figure (e.g., the fraction of MBs with a change rate smaller than 1%).
• Figure 4: Time savings of µOpTime with warmup phase: The x-axis shows every project from data set 2. The left y-axis shows the execution time in hours of the bars, split into warmup and measurement phase. The right y-axis shows different fractions of microbenchmarks (MBs) with low change rates after using µOpTime, represented by marks in the figure (e.g., the fraction of MBs with a change rate smaller than 1%).