Towards an Optimized Benchmarking Platform for CI/CD Pipelines

TL;DR

This vision paper tackles the problem of detecting performance regressions in large-scale, cloud-based software by advocating frequent benchmarking within CI/CD pipelines. It identifies three core challenges—composability of benchmark optimization strategies, automated evaluation of results, and usability in practice—and argues they currently hinder adoption. The authors propose a conceptual cloud-based benchmarking framework that orchestrates benchmark selection, optimization, execution, and automated result analysis, with a data-backed foundation for improving subsequent runs. By outlining research directions and a practical framework, the work aims to make continuous benchmarking more feasible and impactful for reducing resource waste and maintaining SLAs in modern software systems.

Abstract

Performance regressions in large-scale software systems can lead to substantial resource inefficiencies, making their early detection critical. Frequent benchmarking is essential for identifying these regressions and maintaining service-level agreements (SLAs). Performance benchmarks, however, are resource-intensive and time-consuming, which is a major challenge for integration into Continuous Integration / Continuous Deployment (CI/CD) pipelines. Although numerous benchmark optimization techniques have been proposed to accelerate benchmark execution, there is currently no practical system that integrates these optimizations seamlessly into real-world CI/CD pipelines. In this vision paper, we argue that the field of benchmark optimization remains under-explored in key areas that hinder its broader adoption. We identify three central challenges to enabling frequent and efficient benchmarking: (a) the composability of benchmark optimization strategies, (b) automated evaluation of benchmarking results, and (c) the usability and complexity of applying these strategies as part of CI/CD systems in practice. We also introduce a conceptual cloud-based benchmarking framework handling these challenges transparently. By presenting these open problems, we aim to stimulate research toward making performance regression detection in CI/CD systems more practical and effective.

Figures (5)

• Figure 1: A typical cloud benchmarking scenario: The SUT and the corresponding load generator of the application benchmark are deployed on different VMs in the cloud. This ensures that the load generator does not take CPU time away from the SUT by effectively isolating both components.
• Figure 2: Microbenchmarks with RMIT: Here we see an example of microbenchmarks executed using a simple RMIT schema. The microbenchmark suite consists of two microbenchmarks, labeled A and B. Each microbenchmark is executed for two iterations, the suite is executed two times, and everything is replicated across two different cloud instances (i.e., VMs).
• Figure 3: A CI/CD pipeline with a benchmark using dynamic optimization.
• Figure 4: A CI/CD pipeline with a benchmark using static optimization.
• Figure 5: A generic high-level overview of an integrated benchmarking framework for automatically optimizing benchmarks.