Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Synthesis-in-the-Loop Evaluation of LLMs for RTL Generation: Quality, Reliability, and Failure Modes

Weimin Fu, Zeng Wang, Minghao Shao, Ramesh Karri, Muhammad Shafique, Johann Knechtel, Ozgur Sinanoglu, Xiaolong Guo

Abstract

RTL generation demands more than software code synthesis: designs must be syntactically valid, synthesizable, functionally correct, and hardware-efficient. Existing evaluations stop at functional correctness, leaving synthesizability and implementation quality unmeasured. We evaluate 32 language models on 202 Verilog tasks from VerilogEval and RTLLM, with five attempts each, scoring via the Hardware Quality Index (HQI), a 0--100 metric integrating post-synthesis area, delay, and warning count relative to expert references under a Nangate45 45\,nm flow. Three performance tiers emerge: 13 frontier models achieve Global HQI above 71, led by Gemini-3-Pro (87.5\% coverage, 85.1 HQI); 11 mid-tier models cluster at 53--68; 8 fall below 53. The capability-to-deployment gap (best-of-five vs.\ single-attempt) spans 3.8--22.1 HQI points, motivating multi-sample strategies. A tool-adjudicated taxonomy of 195 genuine synthesis failures reveals systematic divergence: proprietary models fail late through elaboration errors and synthesis timeout; open-weight models fail early through missing module wrappers and non-synthesizable constructs, consistent with training on simulation-grade rather than synthesis-grade RTL. Rankings hold across three technology libraries at Spearman~$ρ> 0.99$.

Synthesis-in-the-Loop Evaluation of LLMs for RTL Generation: Quality, Reliability, and Failure Modes

Abstract

RTL generation demands more than software code synthesis: designs must be syntactically valid, synthesizable, functionally correct, and hardware-efficient. Existing evaluations stop at functional correctness, leaving synthesizability and implementation quality unmeasured. We evaluate 32 language models on 202 Verilog tasks from VerilogEval and RTLLM, with five attempts each, scoring via the Hardware Quality Index (HQI), a 0--100 metric integrating post-synthesis area, delay, and warning count relative to expert references under a Nangate45 45\,nm flow. Three performance tiers emerge: 13 frontier models achieve Global HQI above 71, led by Gemini-3-Pro (87.5\% coverage, 85.1 HQI); 11 mid-tier models cluster at 53--68; 8 fall below 53. The capability-to-deployment gap (best-of-five vs.\ single-attempt) spans 3.8--22.1 HQI points, motivating multi-sample strategies. A tool-adjudicated taxonomy of 195 genuine synthesis failures reveals systematic divergence: proprietary models fail late through elaboration errors and synthesis timeout; open-weight models fail early through missing module wrappers and non-synthesizable constructs, consistent with training on simulation-grade rather than synthesis-grade RTL. Rankings hold across three technology libraries at Spearman~.
Paper Structure (20 sections, 3 equations, 3 figures, 1 table)

This paper contains 20 sections, 3 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Methodology
  4. Benchmark and Model Set
  5. Evaluation Pipeline
  6. Hardware Quality Index and Aggregate Metrics
  7. Per-attempt HQI.
  8. Coverage, Global HQI, and Expected HQI.
  9. Results
  10. Overall Capability Landscape
  11. Pass rate versus implementation quality.
  12. Within-Provider Scaling: The OpenAI Lineage
  13. Deployment Reliability: Best-of-Five versus Per-Attempt Quality
  14. Category-Level Analysis
  15. Synthesis Failure Analysis
  16. ...and 5 more sections

Figures (3)

  • Figure 1: Best-of-five HQI heatmap across eight hardware categories and 32 models. Models are ordered left-to-right by Global HQI; categories are ordered top-to-bottom by average Tier 1 accuracy. Color encodes HQI score (0--100).
  • Figure 2: Per-attempt Expected HQI heatmap across eight hardware categories and 32 models, using the same ordering as Figure \ref{['fig:heatmap_best5']}. Scores reflect single-attempt deployment quality rather than the best-of-five capability ceiling.
  • Figure 3: Inference characteristics across all 32 models measured via the OpenRouter API. Cost and TTFT use log scale.