Pluto: A Benchmark for Evaluating Efficiency of LLM-generated Hardware Code

TL;DR

Pluto introduces a benchmark and evaluation framework for assessing the synthesis efficiency of LLM-generated Verilog by providing 114 problems with per-metric ground-truth optimizations (area, delay, power), self-checking testbenches tolerant to latency variations, and three metric-specific reference implementations. It extends the efficiency metric to a three-dimensional space and defines eff@k alongside the standard pass@k to quantify both correctness and hardware efficiency across two problem formulations. Experimental results show that while contemporary LLMs achieve high functional correctness, their synthesis efficiency lags behind expert-crafted designs, underscoring the need for metric-aware benchmarks to guide hardware-focused research in LLMs. Pluto’s ablations confirm robustness across synthesis tools and libraries and illustrate how metric-specific optimizations shift Pareto fronts, highlighting the importance of process and feedback in achieving Pareto-optimal hardware designs. Overall, Pluto provides a rigorous, scalable framework to drive progress toward efficiency-aware LLM-enabled hardware design.

Abstract

Large Language Models (LLMs) are increasingly used to automate hardware design tasks, including the generation of Verilog code. While early benchmarks focus primarily on functional correctness, efficient hardware design demands additional optimization for synthesis metrics such as area, delay, and power. Existing benchmarks fall short in evaluating these aspects comprehensively: they often lack optimized baselines or testbenches for verification. To address these gaps, we present Pluto, a benchmark and evaluation framework designed to assess the efficiency of LLM-generated Verilog designs. Pluto presents a comprehensive evaluation set of 114 problems with self-checking testbenches and multiple Pareto-optimal reference implementations. Experimental results show that state-of-the-art LLMs can achieve high functional correctness, reaching 78.3\% at pass@1, but their synthesis efficiency still lags behind expert-crafted implementations, with area efficiency of 63.8\%, delay efficiency of 65.9\%, and power efficiency of 64.0\% at eff@1. This highlights the need for efficiency-aware evaluation frameworks such as Pluto to drive progress in hardware-focused LLM research.

Figures (8)

• Figure 1: Overview of the Pluto benchmark on the trailing zeros detection task. We show three reference implementations optimized for different synthesis metrics compared to the unoptimized baseline: (left) area, using a mux-based priority encoder, reducing area by 33%; (center) delay, using an LSB isolation circuit with a parallel one-hot encoder, reducing delay by 44%; and (right) power, using an LSB-to-MSB scanning method with early termination, reducing total power by 34%. See Appendix. \ref{['appendix:trailing_zeros']} for unoptimized baseline and self-checking testbench.
• Figure 2: Distribution of area, delay, and power across Pluto benchmark designs before and after manual metric-specific optimizations.
• Figure 3: Efficiency scores for area, delay, and power across all benchmark problems, using both Cadence Genus and Yosys with different technology libraries. Results show consistent efficiency trends across synthesis tools and technologies.
• Figure 4: Area–delay tradeoffs of three problems in the Pluto benchmark under different synthesis strategies. Each strategy corresponds to a distinct sequence of ABC logic synthesis commands.
• Figure 5: Failure mode analysis of optimization outcomes. (a) Quadrant plot showing the correlation between functional correctness (Pass@1) and synthesis efficiency (Eff@1) across area, delay, and power objectives. (b) Heatmap of optimization strategy difficulty across different optimization objectives area, delay, and power.