GPU acceleration of non-equilibrium Green's function calculation using OpenACC and CUDA FORTRAN

TL;DR

The paper presents high-performance implementations of the Kadanoff-Baym equations for nonequilibrium Green's functions on distributed CPUs and multi-GPU systems. It compares MPI/OpenMP on CPUs with GPU-accelerated OpenACC and CUDA FORTRAN approaches, introducing loop-fusion, single-kernel launches, memory-management optimizations, and on-the-fly indexing to maximize GPU throughput. CUDA FORTRAN often outperforms OpenACC due to finer-grained control over thread hierarchy and reductions, achieving substantial speedups while maintaining strong and weak scaling. The findings indicate GPU-enabled NEGF computations are viable for large-scale quantum many-body dynamics, significantly reducing wall-clock times without sacrificing numerical accuracy.

Abstract

The numerical solution of the Kadanoff-Baym nonlinear integro-differential equations, which yields the non-equilibrium Green's functions (NEGFs) of quantum many-body systems, poses significant computational challenges due to its high computational complexity. In this work, we present efficient implementations of a numerical method for solving these equations on distributed-memory architectures, including many-core CPUs and multi-GPU systems. For CPU-based platforms, we adopt a hybrid MPI/OpenMP programming model to exploit both inter-node and intra-node parallelism. On GPU-accelerated systems, we implement the method using two distinct approaches: MPI/OpenACC and MPI/CUDA FORTRAN. Several optimization strategies are employed to enhance GPU performance, including techniques to maximize computational resource utilization and minimize the overhead associated with kernel launches and memory management. Although OpenACC is easy to use, CUDA FORTRAN provides more advanced features for configuring and managing multiple levels of concurrency, while also simplifying memory allocation and data movement between host and device. This flexibility translates into significant performance improvements. We compare the performance of the three implementations and demonstrate that the GPU-based approaches achieve substantial speedups over CPU-based implementations. Furthermore, both CPU and GPU versions exhibit excellent strong and weak scaling, confirming the scalability and efficiency of our approach for large-scale NEGF computations.

Figures (20)

• Figure 1: Illustration of the evaluation at the $n_t$-th time step for $\Sigma^\lessgtr$ and $I^\lessgtr$. The blue dots represent values computed at previous time steps, and the orange dots give the $2n_t-1$ pairs of $(t,t')$ at which the values of $\Sigma^\lessgtr$ and $I^\lessgtr$ are computed at the $n_t$-th time step.
• Figure 2: The main steps in the outermost loop of the KBE solver.
• Figure 3: The MPI implementation of $\Sigma^2_{jm}({\bf k};t,t')$ computation at a given $(t,t')$ pair.
• Figure 4: The hybrid MPI/OPenMP implementation of the $\Sigma^2_{jm}(\mathbf{k}; t,t')$ computation at a given $(t,t')$ pair.
• Figure 5: The performance of the MPI and the hybrid MPI/OpenMP implementation to the program with varying number of MPI ranks. In the test problem, $n_k=1024$, and we compute the first $10$ time steps.