Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CUDABench: Benchmarking LLMs for Text-to-CUDA Generation

Jiace Zhu, Wentao Chen, Qi Fan, Zhixing Ren, Junying Wu, Xing Zhe Chai, Chotiwit Rungrueangwutthinon, Yehan Ma, An Zou

TL;DR

This work introduces CUDABench, a comprehensive benchmark designed to evaluate the text-to-CUDA capabilities of LLMs, and proposes CUDABench-Score and Generative Verification Pipeline that assess compilation correctness, functional consistency through execution-based verification, and a novel roofline-based metric, Performance-Score.

Abstract

Recent studies have demonstrated the potential of Large Language Models (LLMs) in generating GPU Kernels. Current benchmarks focus on the translation of high-level languages into CUDA, overlooking the more general and challenging task of text-to-CUDA generation. Furthermore, given the hardware-specific and performance-critical features of GPU programming, accurately assessing the performance of LLM-generated GPU programs is nontrivial. In this work, we introduce CUDABench, a comprehensive benchmark designed to evaluate the text-to-CUDA capabilities of LLMs. First, we construct CUDABench-Set, which covers Breadth-Depth-Difficulty evaluation space in diverse application domains, including artificial intelligence, scientific computing, and data analytics, etc. Furthermore, we propose CUDABench-Score and Generative Verification Pipeline that assess (1) compilation correctness, (2) functional consistency through execution-based verification, and (3) a novel roofline-based metric, Performance-Score. Benchmarking state-of-the-art LLMs reveals insightful findings and challenges of text-to-CUDA, such as a notable mismatch between high compilation success rates and low functional correctness, a lack of domain-specific algorithmic knowledge, and suboptimal utilization of GPU hardware resources. Our benchmark is available at https://github.com/CUDA-Bench/CUDABench.

CUDABench: Benchmarking LLMs for Text-to-CUDA Generation

TL;DR

This work introduces CUDABench, a comprehensive benchmark designed to evaluate the text-to-CUDA capabilities of LLMs, and proposes CUDABench-Score and Generative Verification Pipeline that assess compilation correctness, functional consistency through execution-based verification, and a novel roofline-based metric, Performance-Score.

Abstract

Recent studies have demonstrated the potential of Large Language Models (LLMs) in generating GPU Kernels. Current benchmarks focus on the translation of high-level languages into CUDA, overlooking the more general and challenging task of text-to-CUDA generation. Furthermore, given the hardware-specific and performance-critical features of GPU programming, accurately assessing the performance of LLM-generated GPU programs is nontrivial. In this work, we introduce CUDABench, a comprehensive benchmark designed to evaluate the text-to-CUDA capabilities of LLMs. First, we construct CUDABench-Set, which covers Breadth-Depth-Difficulty evaluation space in diverse application domains, including artificial intelligence, scientific computing, and data analytics, etc. Furthermore, we propose CUDABench-Score and Generative Verification Pipeline that assess (1) compilation correctness, (2) functional consistency through execution-based verification, and (3) a novel roofline-based metric, Performance-Score. Benchmarking state-of-the-art LLMs reveals insightful findings and challenges of text-to-CUDA, such as a notable mismatch between high compilation success rates and low functional correctness, a lack of domain-specific algorithmic knowledge, and suboptimal utilization of GPU hardware resources. Our benchmark is available at https://github.com/CUDA-Bench/CUDABench.
Paper Structure (25 sections, 5 equations, 6 figures, 3 tables)

This paper contains 25 sections, 5 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Text-to-CUDA Dataset: CUDABench-Set
  4. Breadth.
  5. Depth.
  6. Difficulty.
  7. Text-to-CUDA Metrics: CUDABench-Score
  8. Correctness Metrics
  9. Performance Metrics
  10. Performance-Score
  11. Memory-Bound & Compute-Bound
  12. CUDABench-Score
  13. Generative Verification Pipeline
  14. Experiment
  15. Experimental Setup
  16. ...and 10 more sections

Figures (6)

  • Figure 1: Overview of CUDABench.
  • Figure 2: Statistical distribution of CUDABench-Set dimensions. (a) Breadth: Distribution of tasks across domains. (b) Depth: Input size distribution across scale levels. (c) Difficulty: Prompt length distribution across difficulty levels.
  • Figure 3: The roofline model for an NVIDIA A40 GPU. The $y$-axis is floating-point performance. The $x$-axis is arithmetic intensity. The A40 GPU has peak single-precision floating-point performance of 37.4 TFLOPs/sec and peak memory bandwidth of 696 GB/sec. The chart delineates Memory-Bound (red shaded) and Compute-Bound (blue shaded) regions. It visualizes the performance bottlenecks of an MSE kernel (red dot) and a Conv2D kernel (blue triangle), contrasted with an optimized MSE kernel (green dot) that approaches the theoretical hardware limit.
  • Figure 4: Comparison of Roofline models for the MSE kernel executed on different GPU. Arith denotes Arithmetic Intensity, Perf refers to the Achieved GFLOPs/sec, Peak represents the Attainable GFLOPs/sec, and Score corresponds to the calculated Performance-Score.
  • Figure 5: The proportion of Compilation Failure (Red), Functional Inconsistency (Yellow), and Valid (Green) kernel for each model. Results are broken down by difficulty levels (Pass@1).
  • ...and 1 more figures