Chopper: A Multi-Level GPU Characterization Tool & Derived Insights Into LLM Training Inefficiency

TL;DR

Chopper introduces a multi-granularity GPU profiling framework and demonstrates its application to Llama 3 8B training on an eight-GPU AMD MI300X node under Fully Sharded Data Parallelism. The framework reveals that frequency-related DVFS overhead is the dominant factor limiting end-to-end performance, surpassing MFMA utilization, overlap, and kernel-launch overheads. By enabling end-to-end trace alignment and metric aggregation, Chopper provides actionable insights into operation overlap, launch dynamics, and CPU underutilization, guiding optimizations in training frameworks and hardware design. The work establishes a first holistic characterization of LLM training on MI300X GPUs and highlights frequency management and memory determinism as practical levers for performance and energy efficiency.

Abstract

Training large language models (LLMs) efficiently requires a deep understanding of how modern GPU systems behave under real-world distributed training workloads. While prior work has focused primarily on kernel-level performance or single-GPU microbenchmarks, the complex interaction between communication, computation, memory behavior, and power management in multi-GPU LLM training remains poorly characterized. In this work, we introduce Chopper, a profiling and analysis framework that collects, aligns, and visualizes GPU kernel traces and hardware performance counters across multiple granularities (i.e., from individual kernels to operations, layers, phases, iterations, and GPUs). Using Chopper, we perform a comprehensive end-to-end characterization of Llama 3 8B training under fully sharded data parallelism (FSDP) on an eight-GPU AMD InstinctTM MI300X node. Our analysis reveals several previously underexplored bottlenecks and behaviors, such as memory determinism enabling higher, more stable GPU and memory frequencies. We identify several sources of inefficiencies, with frequency overhead (DVFS effects) being the single largest contributor to the gap between theoretical and observed performance, exceeding the impact of MFMA utilization loss, communication/computation overlap, and kernel launch overheads. Overall, Chopper provides the first holistic, multi-granularity characterization of LLM training on AMD InstinctTM MI300X GPUs, yielding actionable insights for optimizing training frameworks, improving power-management strategies, and guiding future GPU architecture and system design.

Figures (15)

• Figure 1: Diagram of operations in Llama. Colors of operation types match those used in Figure \ref{['fig:end_to_end']}. i_e is input embedding, ln is the final RMSNorm, lp is logits projection, attn/mlp_n is attention/multi-layer perceptron (MLP) RMSNorm, attn/mlp_ra is residual add, attn_fa is FlashAttention, attn_or is output reshape, attn_op is output projection, qkv_ip is the QKV (query, key, value) input projection, qkv_s is split, qkv_t is transpose, qkv_re is rotary embedding, qkv_c is contiguous memory copy, mlp_gp is gate projection, mlp_gs is silu, mlp_up is up projection, mlp_gu is gate-up elementwise multiply, and mlp_dp is down projection.
• Figure 2: Overview of FSDP. Notation: all gather (AG); forward (FWD); backward (BWD); (D) delete extra model weights; reduce scatter (RS); optimize (OPT).
• Figure 3: Overview of the Chopper framework.
• Figure 4: Median values across iterations and GPUs. Top row is normalized to b1s4 with FSDPv1. Duration is the sum of kernel duration. Launch overhead is the sum of bubbles between the previous and current kernel, and corresponds to the chunk or operation beneath it. Throughput is calculated with the maximum duration plus launch overhead across GPUs. Notation: batch size (b), sequence length in K tokens (s), forward pass (fwd), backward pass (bwd), optimization step (opt), FlashAttention (fa), vector operation (vec), matrix multiply (gemm).
• Figure 5: Operation duration for different operator types and model configurations using FSDPv1 and v2. Duration is summed across layers and includes bubbles between the kernels of each operation.