EvoEngineer: Mastering Automated CUDA Kernel Code Evolution with Large Language Models

TL;DR

The paper tackles automated CUDA kernel optimization by introducing EvoEngineer, a systematic framework that splits LLM-driven code evolution into traverse techniques and population management. This decomposition enables principled strategy selection to balance performance and correctness, validated across 91 real-world CUDA kernels. The study demonstrates a median speedup of $2.72\times$ with a code validity of $69.8\%$, and a maximum kernel speedup of $36.75\times$ over PyTorch kernels, with EvoEngineer achieving the highest speedup on a majority of operations. The work provides a modular, open-source platform and empirical guidelines for selecting optimization strategies in real-world GPU kernel optimization tasks, paving the way for more robust and scalable automated performance engineering.

Abstract

CUDA kernel optimization has become a critical bottleneck for AI performance, as deep learning training and inference efficiency directly depends on highly optimized GPU kernels. Despite the promise of Large Language Models (LLMs) for automating kernel optimization, this field suffers from a fragmented ecosystem of isolated and incomparable approaches with unclear problem formulations. Furthermore, general-purpose LLM code evolution methods cannot meet strict correctness requirements of CUDA kernel optimization. We address these fundamental challenges by first formalizing CUDA kernel optimization as a code optimization task with a clear objective, constraints, and evaluation metrics. We then establish the first systematic LLM-based code evolution framework, EvoEngineer, that provides guidance for designing and adapting optimization strategies to achieve a balance between performance and correctness. Finally, we implement a kernel optimization system based on this framework and conduct extensive experiments on 91 real-world CUDA kernels. Our results demonstrate that EvoEngineer achieves a principled balance between performance and correctness, with the highest averaged median speedup of \textbf{2.72}$\times$ over baseline CUDA kernels and a code validity rate of \textbf{69.8}\%, outperforming existing methods on both dimensions. Our method achieves a maximum speedup of \textbf{36.75}$\times$ among all operations over PyTorch kernels and delivers the highest speedup on \textbf{28} (\textbf{56.0\%}) of 50 operations that achieve over \textbf{2$\times$} acceleration.

Figures (9)

• Figure 1: Trade-off between speedup and functional correctness across different optimization strategies, showing the dominance of EvoEngineer variants.
• Figure 2: EvoEngineer framework for LLM-based code evolution. The framework includes two orthogonal components: traverse techniques for navigation and population management for solution maintenance. Traverse techniques consist of a solution guiding layer that defines navigation strategy and a prompt engineering layer that communicates this strategy to the LLM.
• Figure 3: Overview of our modular kernel optimization system. This system follows a three-step workflow: (1) Task Configuration, (2) Solution Generation, and (3) Feedback Collection.
• Figure 4: Token usage analysis for different methods using GPT-4.1.
• Figure 5: Speedup over 2$\times$ compared to PyTorch kernels.