DiffBench Meets DiffAgent: End-to-End LLM-Driven Diffusion Acceleration Code Generation

TL;DR

This paper tackles the latency inherent in diffusion models by introducing DiffBench, a comprehensive benchmark for diffusion-acceleration code generation, and DiffAgent, an LLM-driven framework that plans, codes, debugs, and optimizes diffusion inference code via a genetic-algorithm guided loop. The approach formalizes diffusion-acceleration as an automated, environment-aware coding task, couples a three-stage evaluation pipeline with CLIP-based quality checks and latency/throughput targets, and demonstrates substantial performance gains across multiple base LLMs. Key contributions include the DiffBench benchmark (604 tasks across architectures and deployment scenarios) and the DiffAgent system (planning, coding, debugging, GA selector) that jointly improve correctness and efficiency, reducing common error modes. The work enables automated generation of deployment-ready diffusion acceleration code, enabling practical, scalable acceleration across diverse diffusion architectures and hardware platforms.

Abstract

Diffusion models have achieved remarkable success in image and video generation. However, their inherently multiple step inference process imposes substantial computational overhead, hindering real-world deployment. Accelerating diffusion models is therefore essential, yet determining how to combine multiple model acceleration techniques remains a significant challenge. To address this issue, we introduce a framework driven by large language models (LLMs) for automated acceleration code generation and evaluation. First, we present DiffBench, a comprehensive benchmark that implements a three stage automated evaluation pipeline across diverse diffusion architectures, optimization combinations and deployment scenarios. Second, we propose DiffAgent, an agent that generates optimal acceleration strategies and codes for arbitrary diffusion models. DiffAgent employs a closed-loop workflow in which a planning component and a debugging component iteratively refine the output of a code generation component, while a genetic algorithm extracts performance feedback from the execution environment to guide subsequent code refinements. We provide a detailed explanation of the DiffBench construction and the design principles underlying DiffAgent. Extensive experiments show that DiffBench offers a thorough evaluation of generated codes and that DiffAgent significantly outperforms existing LLMs in producing effective diffusion acceleration strategies.

Figures (5)

• Figure 1: Overview of our proposed DiffBench. DiffBench tasks LLMs with generating diffusion acceleration code from a given user prompt and performs automated three-step evaluation.
• Figure 2: Illustration of the pipeline and distribution of acceleration methods. Our benchmark covers diverse pipelines and acceleration strategies to enable comprehensive evaluation of LLM-driven acceleration code generation.
• Figure 3: Overview of our proposed DiffAgent. Through multi-agent collaboration and genetic algorithm based optimization, DiffAgent generates high-quality diffusion acceleration code and iteratively refines it.
• Figure 4: Error mode analysis on the DiffBench benchmark across various LLMs, reported per difficulty level. We categorize failure modes of generated code into five failure models. All values are percentages.
• Figure 5: Sample visualization of code generation results. Incorrect code segments originally generated by the LLM are highlighted in red, while corrected segments produced by DiffAgent are shown in green.