CUDA-LLM: LLMs Can Write Efficient CUDA Kernels

TL;DR

The paper tackles the challenge of generating high-performance CUDA kernels with LLMs by introducing the Feature Search and Reinforcement (FSR) framework, which jointly optimizes functional correctness and runtime performance through a multi-prompt, reinforcement-driven loop. FSR uses three feature functions—Compilation Verifier, Function Validator, and Performance Profiler—to iteratively refine candidate kernels toward hardware-specific efficiency, validated on 20 CUDA kernels across edge and server GPUs. Empirical results show that CUDA-LLM with FSR achieves correctness on all tasks while delivering substantial speedups compared to directly generated kernels, with notable improvements up to 179x (and some cases >300x) and robust numerical accuracy. The work demonstrates a practical path to automated, architecture-aware GPU kernel generation, with potential to transform GPU programming tasks across diverse hardware platforms.

Abstract

Large Language Models (LLMs) have demonstrated strong capabilities in general-purpose code generation. However, generating the code which is deeply hardware-specific, architecture-aware, and performance-critical, especially for massively parallel GPUs, remains a complex challenge. In this work, we explore the use of LLMs for the automated generation and optimization of CUDA programs, with the goal of producing high-performance GPU kernels that fully exploit the underlying hardware. To address this challenge, we propose a novel framework called \textbf{Feature Search and Reinforcement (FSR)}. FSR jointly optimizes compilation and functional correctness, as well as the runtime performance, which are validated through extensive and diverse test cases, and measured by actual kernel execution latency on the target GPU, respectively. This approach enables LLMs not only to generate syntactically and semantically correct CUDA code but also to iteratively refine it for efficiency, tailored to the characteristics of the GPU architecture. We evaluate FSR on representative CUDA kernels, covering AI workloads and computational intensive algorithms. Our results show that LLMs augmented with FSR consistently guarantee correctness rates. Meanwhile, the automatically generated kernels can outperform general human-written code by a factor of up to 179$\times$ in execution speeds. These findings highlight the potential of combining LLMs with performance reinforcement to automate GPU programming for hardware-specific, architecture-sensitive, and performance-critical applications.

Figures (4)

• Figure 1: Overview of Feature Search and Reinforcement (FSR) framework.
• Figure 2: The Search and Reinforcement Process.
• Figure 3: Performance evaluation across various tasks.
• Figure :