Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GPU Acceleration of Monte Carlo Tallies on Unstructured Meshes in OpenMC with PUMI-Tally

Fuad Hasan, Cameron W. Smith, Mark S. Shephard, R. Michael Churchill, George J. Wilkie, Paul K. Romano, Patrick C. Shriwise, Jacob S. Merson

TL;DR

This paper addresses the bottleneck of unstructured-mesh tallies in fusion-relevant Monte Carlo neutronics by introducing PUMI-Tally, a GPU-accelerated tally library that exploits mesh adjacency for fast, batched tally operations on CPU and GPU. Implemented as a PIMPL-based extension to OpenMC, it decouples dependency on GPU-compilers and leverages adjacency-based ray tracing with the PUMI-Pic framework to perform tallies during particle transport. The approach achieves large performance gains (up to ~20X speedups), dramatically reduces memory allocations (≈199X fewer), and delivers energy savings in hybrid CPU/GPU configurations, validating both correctness against a baseline OpenMC and practical scalability for large, unstructured meshes. The results support substantial practical impact for fusion neutron transport simulations, with asynchronous GPU tallying and CPU/GPU coexistence enabling efficient workflows and opportunities for further GPU-only transport-tally integration.

Abstract

Unstructured mesh tallies are a bottleneck in Monte Carlo neutral particle transport simulations of fusion reactors. This paper introduces the PUMI-Tally library that takes advantage of mesh adjacency information to accelerate these tallies on CPUs and GPUs. For a fixed source simulation using track-length tallies, we achieved a speed-up of 19.7X on an NVIDIA A100, and 9.2X using OpenMP on 128 threads of two AMD EPYC 7763 CPUs on NERSC Perlmutter. On the Empire AI alpha system, we achieved a speed-up of 20X using an NVIDIA H100 and 96 threads of an Intel Xenon 8568Y+. Our method showed better scaling with number of particles and number of elements. Additionally, we observed a 199X reduction in the number of allocations during initialization and the first three iterations, with a similar overall memory consumption. And, our hybrid CPU/GPU method demonstrated a 6.69X improvement in the energy consumption over the current approach.

GPU Acceleration of Monte Carlo Tallies on Unstructured Meshes in OpenMC with PUMI-Tally

TL;DR

This paper addresses the bottleneck of unstructured-mesh tallies in fusion-relevant Monte Carlo neutronics by introducing PUMI-Tally, a GPU-accelerated tally library that exploits mesh adjacency for fast, batched tally operations on CPU and GPU. Implemented as a PIMPL-based extension to OpenMC, it decouples dependency on GPU-compilers and leverages adjacency-based ray tracing with the PUMI-Pic framework to perform tallies during particle transport. The approach achieves large performance gains (up to ~20X speedups), dramatically reduces memory allocations (≈199X fewer), and delivers energy savings in hybrid CPU/GPU configurations, validating both correctness against a baseline OpenMC and practical scalability for large, unstructured meshes. The results support substantial practical impact for fusion neutron transport simulations, with asynchronous GPU tallying and CPU/GPU coexistence enabling efficient workflows and opportunities for further GPU-only transport-tally integration.

Abstract

Unstructured mesh tallies are a bottleneck in Monte Carlo neutral particle transport simulations of fusion reactors. This paper introduces the PUMI-Tally library that takes advantage of mesh adjacency information to accelerate these tallies on CPUs and GPUs. For a fixed source simulation using track-length tallies, we achieved a speed-up of 19.7X on an NVIDIA A100, and 9.2X using OpenMP on 128 threads of two AMD EPYC 7763 CPUs on NERSC Perlmutter. On the Empire AI alpha system, we achieved a speed-up of 20X using an NVIDIA H100 and 96 threads of an Intel Xenon 8568Y+. Our method showed better scaling with number of particles and number of elements. Additionally, we observed a 199X reduction in the number of allocations during initialization and the first three iterations, with a similar overall memory consumption. And, our hybrid CPU/GPU method demonstrated a 6.69X improvement in the energy consumption over the current approach.
Paper Structure (7 sections, 3 equations, 9 figures, 3 algorithms)

This paper contains 7 sections, 3 equations, 9 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Implementation
  4. Integration into OpenMC
  5. Problem Setup and Verification
  6. Results
  7. Conclusion

Figures (9)

  • Figure 1: Adjacency based search algorithm.
  • Figure 2: Simulation domain for verification problem. The red cube represents the location of the neutron source.
  • Figure 3: Comparison of flux tallies from current OpenMC and our version of OpenMC utilizing PUMI-Tally.
  • Figure 4: Runtime vs. number of particles for 10,000 elements (a), 100,000 elements (b) 500,000 elements (c). Colors show the proportion of the solution time spent in search initialization (blue), active batch, i.e., transport and tally (red), data copies to the GPU (orange), and other operations (green).
  • Figure 5: Runtime to simulate 800,000 and 2 million particles vs. the number of elements for the current solution (red circles) and PUMI-Tally (green squares).
  • ...and 4 more figures