Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CI/CD Efforts for Validation, Verification and Benchmarking OpenMP Implementations

Aaron Jarmusch, Felipe Cabarcas, Swaroop Pophale, Andrew Kallai, Johannes Doerfert, Luke Peyralans, Seyong Lee, Joel Denny, Sunita Chandrasekaran

TL;DR

HPC exascale systems demand reliable OpenMP offloading across diverse compilers and GPUs. The authors implement a CI/CD workflow that integrates the OpenMP Validation & Verification (OMPVV) suite and SPEChpc benchmarks to continuously assess OpenMP offloading support across LLVM Clang/Flang, GCC, NVIDIA, AMD, and Intel toolchains on Frontier, Perlmutter, and Sunspot. Key contributions include the pipeline design for nightly testing, cross-compiler validation results, and benchmarking insights that reveal variability by compiler, application, and hardware. This workflow provides rapid feedback to compiler developers, clarifies OpenMP offloading capabilities on heterogeneous platforms, and informs future tooling, optimization, and standardization efforts in HPC software.

Abstract

Software developers must adapt to keep up with the changing capabilities of platforms so that they can utilize the power of High- Performance Computers (HPC), including exascale systems. OpenMP, a directive-based parallel programming model, allows developers to include directives to existing C, C++, or Fortran code to allow node level parallelism without compromising performance. This paper describes our CI/CD efforts to provide easy evaluation of the support of OpenMP across different compilers using existing testsuites and benchmark suites on HPC platforms. Our main contributions include (1) the set of a Continuous Integration (CI) and Continuous Development (CD) workflow that captures bugs and provides faster feedback to compiler developers, (2) an evaluation of OpenMP (offloading) implementations supported by AMD, HPE, GNU, LLVM, and Intel, and (3) evaluation of the quality of compilers across different heterogeneous HPC platforms. With the comprehensive testing through the CI/CD workflow, we aim to provide a comprehensive understanding of the current state of OpenMP (offloading) support in different compilers and heterogeneous platforms consisting of CPUs and GPUs from NVIDIA, AMD, and Intel.

CI/CD Efforts for Validation, Verification and Benchmarking OpenMP Implementations

TL;DR

HPC exascale systems demand reliable OpenMP offloading across diverse compilers and GPUs. The authors implement a CI/CD workflow that integrates the OpenMP Validation & Verification (OMPVV) suite and SPEChpc benchmarks to continuously assess OpenMP offloading support across LLVM Clang/Flang, GCC, NVIDIA, AMD, and Intel toolchains on Frontier, Perlmutter, and Sunspot. Key contributions include the pipeline design for nightly testing, cross-compiler validation results, and benchmarking insights that reveal variability by compiler, application, and hardware. This workflow provides rapid feedback to compiler developers, clarifies OpenMP offloading capabilities on heterogeneous platforms, and informs future tooling, optimization, and standardization efforts in HPC software.

Abstract

Software developers must adapt to keep up with the changing capabilities of platforms so that they can utilize the power of High- Performance Computers (HPC), including exascale systems. OpenMP, a directive-based parallel programming model, allows developers to include directives to existing C, C++, or Fortran code to allow node level parallelism without compromising performance. This paper describes our CI/CD efforts to provide easy evaluation of the support of OpenMP across different compilers using existing testsuites and benchmark suites on HPC platforms. Our main contributions include (1) the set of a Continuous Integration (CI) and Continuous Development (CD) workflow that captures bugs and provides faster feedback to compiler developers, (2) an evaluation of OpenMP (offloading) implementations supported by AMD, HPE, GNU, LLVM, and Intel, and (3) evaluation of the quality of compilers across different heterogeneous HPC platforms. With the comprehensive testing through the CI/CD workflow, we aim to provide a comprehensive understanding of the current state of OpenMP (offloading) support in different compilers and heterogeneous platforms consisting of CPUs and GPUs from NVIDIA, AMD, and Intel.
Paper Structure (9 sections, 4 figures, 7 tables)

This paper contains 9 sections, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background & Motivation
  3. Related Work
  4. Continuous Integration and Continuous Development Workflow
  5. CI/CD Pipeline
  6. OpenMP Validation & Verification Suite and SPEChpc Benchmarking Suite
  7. OpenMP Validation and Verification Suite
  8. SPEChpc HPG Benchmarking Suite
  9. Conclusion

Figures (4)

  • Figure 1: CI/CD Workflow for LLVM Clang and Flang
  • Figure 2: The CI/CD output for a LLVM Clang and Flang pipeline. The green check-mark marks the job as "Pass" while the red 'x' marked the job as "Fail".
  • Figure 3: No. of tests passing per system out of total 477 C/C++ & 265 Fortran tests
  • Figure 4: Evolution of compilers on Frontier and Perlmutter