Can Large Language Models Predict Parallel Code Performance?

TL;DR

This work investigates hardware-free prediction of GPU kernel performance by recasting Roofline-based bottleneck classification into a binary $BB$ vs $CB$ problem and evaluating LLMs on source code plus hardware specs. Using a ground-truth dataset of $340$ CUDA/OpenMP kernels profiled on an RTX $3080$, the study compares four evaluation settings (profiling data, zero-shot, few-shot, and fine-tuning) and distinguishes reasoning from non-reasoning LLMs. When profiling data is available, state-of-the-art LLMs reach near-perfect accuracy; in zero-shot and few-shot scenarios, reasoning-capable LLMs achieve up to about $64\%$ accuracy, indicating meaningful but improvable performance without hardware measurements. Fine-tuning on a small dataset performed poorly, highlighting data requirements for robust transfer. Overall, LLMs show promise for guiding optimization in the absence of runtime profiling and could enable better performance portability across GPUs with improved datasets and prompting strategies.

Abstract

Accurate determination of the performance of parallel GPU code typically requires execution-time profiling on target hardware -- an increasingly prohibitive step due to limited access to high-end GPUs. This paper explores whether Large Language Models (LLMs) can offer an alternative approach for GPU performance prediction without relying on hardware. We frame the problem as a roofline classification task: given the source code of a GPU kernel and the hardware specifications of a target GPU, can an LLM predict whether the GPU kernel is compute-bound or bandwidth-bound? For this study, we build a balanced dataset of 340 GPU kernels, obtained from HeCBench benchmark and written in CUDA and OpenMP, along with their ground-truth labels obtained via empirical GPU profiling. We evaluate LLMs across four scenarios: (1) with access to profiling data of the kernel source, (2) zero-shot with source code only, (3) few-shot with code and label pairs, and (4) fine-tuned on a small custom dataset. Our results show that state-of-the-art LLMs have a strong understanding of the Roofline model, achieving 100% classification accuracy when provided with explicit profiling data. We also find that reasoning-capable LLMs significantly outperform standard LLMs in zero- and few-shot settings, achieving up to 64% accuracy on GPU source codes, without profiling information. Lastly, we find that LLM fine-tuning will require much more data than what we currently have available. This work is among the first to use LLMs for source-level roofline performance prediction via classification, and illustrates their potential to guide optimization efforts when runtime profiling is infeasible. Our findings suggest that with better datasets and prompt strategies, LLMs could become practical tools for HPC performance analysis and performance portability.

Figures (4)

• Figure 1: RTX 3080 Roofline Model and profiled HeCBench executable metrics. CB and BB classifications are shown for DP-FLOP balance point.
• Figure 2: Training/Validation set token count distributions, showing a reasonably balanced dataset.
• Figure 3: Prompt for RQ1. Highlighted text indicates values changed between invocations.
• Figure 4: System prompt to query LLMs. Highlighted text changes based on the queried source code and hardware.