Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

exaCB: Reproducible Continuous Benchmark Collections at Scale Leveraging an Incremental Approach

Jayesh Badwaik, Mathis Bode, Michal Rajski, Andreas Herten

Abstract

The increasing heterogeneity of high-performance computing (HPC) systems and the transition to exascale architectures require systematic and reproducible performance evaluation across diverse workloads. While continuous integration (CI) ensures functional correctness in software engineering, performance and energy efficiency in HPC are typically evaluated outside CI workflows, motivating continuous benchmarking (CB) as a complementary approach. Integrating benchmarking into CI workflows enables reproducible evaluation, early detection of regressions, and continuous validation throughout the software development lifecycle. We present exaCB, a framework for continuous benchmarking developed in the context of the JUPITER exascale system. exaCB enables application teams to integrate benchmarking into their workflows while supporting large-scale, system-wide studies through reusable CI/CD components, established harnesses, and a shared reporting protocol. The framework supports incremental adoption, allowing benchmarks to be onboarded easily and to evolve from basic runnability to more advanced instrumentation and reproducibility. The approach is demonstrated in JUREAP, the early-access program for JUPITER, where exaCB enabled continuous benchmarking of over 70 applications at varying maturity levels, supporting cross-application analysis, performance tracking, and energy-aware studies. These results illustrate the practicality using exaCB for continuous benchmarking for exascale HPC systems across large, diverse collections of scientific applications.

exaCB: Reproducible Continuous Benchmark Collections at Scale Leveraging an Incremental Approach

Abstract

The increasing heterogeneity of high-performance computing (HPC) systems and the transition to exascale architectures require systematic and reproducible performance evaluation across diverse workloads. While continuous integration (CI) ensures functional correctness in software engineering, performance and energy efficiency in HPC are typically evaluated outside CI workflows, motivating continuous benchmarking (CB) as a complementary approach. Integrating benchmarking into CI workflows enables reproducible evaluation, early detection of regressions, and continuous validation throughout the software development lifecycle. We present exaCB, a framework for continuous benchmarking developed in the context of the JUPITER exascale system. exaCB enables application teams to integrate benchmarking into their workflows while supporting large-scale, system-wide studies through reusable CI/CD components, established harnesses, and a shared reporting protocol. The framework supports incremental adoption, allowing benchmarks to be onboarded easily and to evolve from basic runnability to more advanced instrumentation and reproducibility. The approach is demonstrated in JUREAP, the early-access program for JUPITER, where exaCB enabled continuous benchmarking of over 70 applications at varying maturity levels, supporting cross-application analysis, performance tracking, and energy-aware studies. These results illustrate the practicality using exaCB for continuous benchmarking for exascale HPC systems across large, diverse collections of scientific applications.
Paper Structure (32 sections, 9 figures, 1 table)

This paper contains 32 sections, 9 figures, 1 table.

Table of Contents

  1. Introduction
  2. Challenges of HPC Workloads
  3. Collaborative benchmarking
  4. Continuous Benchmarking
  5. A Continuous Benchmarking Example with ExaCB
  6. Use Case: Logistic Map Application
  7. Benchmark: Logmap Benchmark in JUBE
  8. ExaCB Framework Integration
  9. Design Principles and Related Work
  10. Components
  11. Benchmark Repositories
  12. Protocol
  13. Structure
  14. Benchmark Execution
  15. Data Collection
  16. ...and 17 more sections

Figures (9)

  • Figure 1: Collaborative benchmarking methodology for modern HPC systems
  • Figure 2: Categorization of benchmark collections and harness integration.
  • Figure 3: Post-processing orchestrator plotting results of BabelStream (GPU) over time.
  • Figure 4: Post-processing orchestrator plotting results of GRAPH500 over time.
  • Figure 5: Post-processing orchestrator plotting results showing comparison of benchmark results between JEDI, JUWELS Booster and JURECA-DC.
  • ...and 4 more figures