FlashInfer-Bench: Building the Virtuous Cycle for AI-driven LLM Systems

TL;DR

FlashInfer-Bench addresses the challenge of turning AI-generated GPU kernels into production-ready components for large-scale LLM inference by establishing a closed-loop workflow. It defines FlashInfer Trace, curates a real-world workload dataset, and builds a robustness-aware benchmark with isolation and dynamic kernel substitution (apply()) to deploy the best kernels into engines like SGLang and vLLM. The study reveals key insights: compilation is a primary failure mode, hardware intrinsics are underutilized by agents, and high-level DSLs often yield better practicality while still enabling strong end-to-end gains through production-ready substitution. Collectively, the framework provides a practical, reproducible pathway for continual kernel improvement and integration into large-scale AI inference systems.

Abstract

Recent advances show that large language models (LLMs) can act as autonomous agents capable of generating GPU kernels, but integrating these AI-generated kernels into real-world inference systems remains challenging. FlashInfer-Bench addresses this gap by establishing a standardized, closed-loop framework that connects kernel generation, benchmarking, and deployment. At its core, FlashInfer Trace provides a unified schema describing kernel definitions, workloads, implementations, and evaluations, enabling consistent communication between agents and systems. Built on real serving traces, FlashInfer-Bench includes a curated dataset, a robust correctness- and performance-aware benchmarking framework, a public leaderboard to track LLM agents' GPU programming capabilities, and a dynamic substitution mechanism (apply()) that seamlessly injects the best-performing kernels into production LLM engines such as SGLang and vLLM. Using FlashInfer-Bench, we further evaluate the performance and limitations of LLM agents, compare the trade-offs among different GPU programming languages, and provide insights for future agent design. FlashInfer-Bench thus establishes a practical, reproducible pathway for continuously improving AI-generated kernels and deploying them into large-scale LLM inference.

Figures (8)

• Figure 1: FlashInfer-Bench architecture. FlashInfer Trace provides a standard schema for specifying kernel contracts and semantics, and communicating implementations and evaluation results; FlashInfer-Bench Dataset curates production LLM serving workloads; and flashinfer_bench.apply() deploys the fastest validated implementation directly into LLM inference engines.
• Figure 2: FlashInfer Trace schema design. Definition describes the kernel task. Workload describes the real-world input to the kernel. Solution describes the AI-generated solution. Evaluation describes the evaluation result from benchmarking. Each component also includes auxiliary fields that are useful for grouping and filtering.
• Figure 3: FlashInfer-Bench Dataset collection workflow. We serve the major models against real-world traffic with default, common config, and curate the kernel definitions and workloads.
• Figure 4: FlashInfer-Bench Leaderboard. The top performing models at $\text{fast}_{0.95}$ are gemini-2.5-pro, gpt-o3, and gpt-5-2025-08-07. The top performing models in terms of correctness are gpt-5-2025-08-07 (83.9% pass), gpt-o3 (71.3% pass), and gemini-2.5-pro (48.8% pass).
• Figure 5: Workflow of flashinfer_bench.apply(). It is a dynamic dispatcher that retrieves the best Solution with the kernel input at runtime and returns the execution result.