Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Detection of Performance Changes in MooBench Results Using Nyrkiö on GitHub Actions

Shinhyung Yang, David Georg Reichelt, Henrik Ingo, Wilhelm Hasselbring

TL;DR

The paper tackles the problem of detecting performance changes in MooBench measurements within GitHub Actions CI workflows. It proposes integrating the Nyrkiö change-detection service, which uses the E-Divisive Means algorithm, into the MooBench GitHub workflow to automatically identify regression points across benchmark runs. The authors demonstrate a major regression linked to a Linux kernel version change, reproduce the result on both GitHub-hosted and self-hosted runners, and validate that the regression is not caused by software changes. This work provides a practical, reproducible approach for CI/CD-based performance monitoring and regression detection, with potential to extend to other benchmarking scenarios and tracing frameworks.

Abstract

In GitHub with its 518 million hosted projects, performance changes within these projects are highly relevant to the project's users. Although performance measurement is supported by GitHub CI/CD, performance change detection is a challenging topic. In this paper, we demonstrate how we incorporated Nyrkiö to MooBench. Prior to this work, Moobench continuously ran on GitHub virtual machines, measuring overhead of tracing agents, but without change detection. By adding the upload of the measurements to the Nyrkiö change detection service, we made it possible to detect performance changes. We identified one major performance regression and examined the performance change in depth. We report that (1) it is reproducible with GitHub actions, and (2) the performance regression is caused by a Linux Kernel version change.

Detection of Performance Changes in MooBench Results Using Nyrkiö on GitHub Actions

TL;DR

The paper tackles the problem of detecting performance changes in MooBench measurements within GitHub Actions CI workflows. It proposes integrating the Nyrkiö change-detection service, which uses the E-Divisive Means algorithm, into the MooBench GitHub workflow to automatically identify regression points across benchmark runs. The authors demonstrate a major regression linked to a Linux kernel version change, reproduce the result on both GitHub-hosted and self-hosted runners, and validate that the regression is not caused by software changes. This work provides a practical, reproducible approach for CI/CD-based performance monitoring and regression detection, with potential to extend to other benchmarking scenarios and tracing frameworks.

Abstract

In GitHub with its 518 million hosted projects, performance changes within these projects are highly relevant to the project's users. Although performance measurement is supported by GitHub CI/CD, performance change detection is a challenging topic. In this paper, we demonstrate how we incorporated Nyrkiö to MooBench. Prior to this work, Moobench continuously ran on GitHub virtual machines, measuring overhead of tracing agents, but without change detection. By adding the upload of the measurements to the Nyrkiö change detection service, we made it possible to detect performance changes. We identified one major performance regression and examined the performance change in depth. We report that (1) it is reproducible with GitHub actions, and (2) the performance regression is caused by a Linux Kernel version change.

Paper Structure

This paper contains 13 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Continuous Benchmarking
  4. E-Divisive Means Algorithm in Nyrkiö
  5. Performance Change Examination
  6. Previously Detected Changes
  7. Experimental Setup
  8. Statistical Analysis
  9. Evaluation method:
  10. Validation:
  11. Related Work
  12. Conclusion and Future Work
  13. Acknowledgment

Figures (3)

  • Figure 1: Our investigation focuses on the change between cf3c08d and 59d51e3 ( ). Each data point represents the execution time of a Benchmark run. Values $>$ 4500ns are clipped and the maximum value 35624.0 is caused by development code changes.
  • Figure 2: Nyrkiö's detected performance changes
  • Figure 3: Measurements at commits cf3c08d ( ) and 59d51e3 ( ). The boxplot shows the distribution of execution time measurements per runtime environment.