Benchmarking and Dissecting the Nvidia Hopper GPU Architecture

TL;DR

This paper presents a comprehensive, instruction-level benchmarking study of Nvidia Hopper, Ada, and Ampere GPUs to reveal Hopper's microarchitectural characteristics, with a focus on FP8 Tensor Cores, DPX, and distributed shared memory. It combines traditional memory- and Tensor Core–level latency/throughput benchmarks with Transformer Engine–specific evaluations and Hopper-specific features, including DPX and advanced asynchronous data movement. Key contributions include a detailed analysis of Hopper’s WGMMA programming model, FP8 support, and the Transformer Engine’s FP8 performance for Linear and TransformerLayer modules, as well as practical insights into DSM and asynchronous data transfers for AI workloads. The findings demonstrate Hopper’s memory bandwidth and TC efficiency gains, the necessity of using WGMMA to unlock fourth-generation Tensor Core capabilities, and the potential of DPX and DSM to improve GPU programming and performance modeling for large-scale AI systems.

Abstract

Graphics processing units (GPUs) are continually evolving to cater to the computational demands of contemporary general-purpose workloads, particularly those driven by artificial intelligence (AI) utilizing deep learning techniques. A substantial body of studies have been dedicated to dissecting the microarchitectural metrics characterizing diverse GPU generations, which helps researchers understand the hardware details and leverage them to optimize the GPU programs. However, the latest Hopper GPUs present a set of novel attributes, including new tensor cores supporting FP8, DPX, and distributed shared memory. Their details still remain mysterious in terms of performance and operational characteristics. In this research, we propose an extensive benchmarking study focused on the Hopper GPU. The objective is to unveil its microarchitectural intricacies through an examination of the new instruction-set architecture (ISA) of Nvidia GPUs and the utilization of new CUDA APIs. Our approach involves two main aspects. Firstly, we conduct conventional latency and throughput comparison benchmarks across the three most recent GPU architectures, namely Hopper, Ada, and Ampere. Secondly, we delve into a comprehensive discussion and benchmarking of the latest Hopper features, encompassing the Hopper DPX dynamic programming (DP) instruction set, distributed shared memory, and the availability of FP8 tensor cores. The microbenchmarking results we present offer a deeper understanding of the novel GPU AI function units and programming features introduced by the Hopper architecture. This newfound understanding is expected to greatly facilitate software optimization and modeling efforts for GPU architectures. To the best of our knowledge, this study makes the first attempt to demystify the tensor core performance and programming instruction sets unique to Hopper GPUs.

Figures (9)

• Figure 1: Hopper architecture
• Figure 2: The mma and wgmma instructions that perform $D = A \times B + C$ and $D = A \times B \{+ D\}$, respectively.
• Figure 3: Proportion of execution time for different operators when performing FP8 matrix multiplication using te.Linear.
• Figure 4: Comparison of throughput for matrix multiplication with two same-size matrices $D(N*N) = A(N*N) \times B(N*N)$ in different hardware configurations and data types using te.Linear.
• Figure 5: Comparison of latency for the same input text in different hardware configurations and data types using te.TransformerLayer.