VeriMind: Agentic LLM for Automated Verilog Generation with a Novel Evaluation Metric

TL;DR

VeriMind introduces an agentic LLM framework for automated Verilog generation that uses a structured, multi-agent reasoning workflow to iteratively refine code and testbenches. By integrating dedicated agents for prompt refinement, code generation, testbench creation, and verification, the approach improves both syntactic and functional correctness while reducing refinement cycles. A new evaluation metric, pass@ARC, combines success rate with average refinement cycles to provide a more nuanced assessment of iterative code generation. Experimental results on VerilogEval indicate up to 8.3% gains in pass@k and 8.1% gains in pass@ARC, highlighting VeriMind's potential to advance automated RTL development and digital system synthesis.

Abstract

Designing Verilog modules requires meticulous attention to correctness, efficiency, and adherence to design specifications. However, manually writing Verilog code remains a complex and time-consuming task that demands both expert knowledge and iterative refinement. Leveraging recent advancements in large language models (LLMs) and their structured text generation capabilities, we propose VeriMind, an agentic LLM framework for Verilog code generation that significantly automates and optimizes the synthesis process. Unlike traditional LLM-based code generators, VeriMind employs a structured reasoning approach: given a user-provided prompt describing design requirements, the system first formulates a detailed train of thought before the final Verilog code is generated. This multi-step methodology enhances interpretability, accuracy, and adaptability in hardware design. In addition, we introduce a novel evaluation metric-pass@ARC-which combines the conventional pass@k measure with Average Refinement Cycles (ARC) to capture both success rate and the efficiency of iterative refinement. Experimental results on diverse hardware design tasks demonstrated that our approach achieved up to $8.3\%$ improvement on pass@k metric and $8.1\%$ on pass@ARC metric. These findings underscore the transformative potential of agentic LLMs in automated hardware design, RTL development, and digital system synthesis.

Figures (4)

• Figure 1: Proposed Overall Architecture.
• Figure 2: Surface plot of the pass@ARC metric: $z = x \cdot e^{-0.01 \times (y-1)^2}$, where $x$ is the Pass Rate and $y$ is ARC.
• Figure 3: Bar chart showing Pass Rate (x-axis), ARC (y-axis), and pass@ARC (z-axis) comparing the results of VeriMind and other SOTA methods.
• Figure :