GaDE -- GPU-acceleration of time-dependent Dirac Equation for exascale

TL;DR

GaDE introduces a GPU-accelerated solver for the $3D$ time-dependent Dirac equation, enabling large-scale, relativistic electron dynamics simulations on exascale HPC systems. It combines MPI with CUDA/HIP, uses a dual-kinetic-balance $B$-spline radial basis, and employs Crank-Nicholson time propagation with a robust preconditioned BiCGSTAB solver to maintain stability and efficiency on distributed GPUs. The work demonstrates strong single-GPU performance across NVIDIA and AMD architectures and excellent weak scaling up to 2048 GPUs on the LUMI system, with 85% efficiency, alongside meaningful physical validation against reduced models and convergence benchmarks. The open-source GaDE framework thus offers a portable, scalable platform for predictive simulations of relativistic light–matter interactions, with potential extensions to relativistic TDDFT and attosecond/zeptosecond regimes.

Abstract

Modern heterogeneous high-performance computing (HPC) systems powered by advanced graphics processing unit (GPU) architectures enable accelerating computing with unprecedented performance and scalability. Here, we present a GPU-accelerated solver for the three-dimensional (3D) time-dependent Dirac equation optimized for distributed HPC systems. The solver named GaDE is designed to simulate the electron dynamics in atoms induced by electromagnetic fields in the relativistic regime. It combines MPI with CUDA/HIP to target both NVIDIA and AMD GPU architectures. We discuss our implementation strategies in which most of the computations are carried out on GPUs, taking advantage of the GPU-aware MPI feature to optimize communication performance. We evaluate GaDE on the pre-exascale supercomputer LUMI, powered by AMD MI250X GPUs and HPE's Slingshot interconnect. Single-GPU performance on NVIDIA A100, GH200, and AMD MI250X shows comparable performance between A100 and MI250X in compute and memory bandwidth, with GH200 delivering higher performance. Weak scaling on LUMI demonstrates exceptional scalability, achieving 85% parallel efficiency across 2048 GPUs, while strong scaling delivers a 16x speedup on 32 GPUs - 50% efficiency for a communication-intensive, time-dependent Dirac equation solver. These results demonstrate GaDE's high scalability, making it suitable for exascale systems and enabling predictive simulations for ultra-intense laser experiments probing relativistic quantum effects.

Figures (9)

• Figure 1: Zoomed-in view of radial block matrix for $N=110$ 7th-order splines, chosen to make the band structure more clearly visible. Off-diagonal bands connect the positive- and negative energy components.
• Figure 2: The number of non-zero elements (nnz, corresponding to memory requirement) as a function of $\kappa$. Note that for each $\kappa$, the values of $\mu$ is determined by $-|\kappa| < \mu < |\kappa|$.
• Figure 3: UML diagram showing the high-level structure of the GaDE solver, and how tasks are distributed through the hardware: Disk activity is limited to loading pre-generated data and saving the state vector at checkpoints, while computationally intensive linear algebra computations are handled by the GPUs.
• Figure 4: Non-zero structure of $\bm{H}$: Original matrix (Left-hand side) and matrix partitioned between 8 ranks (right-hand side). Shaded part: matrix part owned by an example partition. Green (local) and red (non-local) vector entries referenced during SpMV.
• Figure 5: Single GPU performance comparison between different GPU architectures: NVIDIA A100 GPU, NVIDIA GH200 GPU and AMD MI250X GCD. The results from the 2-socket AMD EPYC 7742 system are also displayed. (Left-hand side) Elapsed wall time per step (s) as a function of $\kappa$. (Right-hand side) Fraction of peak memory bandwidths as a function of $\kappa$.