Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MetRex: A Benchmark for Verilog Code Metric Reasoning Using LLMs

Manar Abdelatty, Jingxiao Ma, Sherief Reda

TL;DR

MetRex introduces a large-scale Verilog metric reasoning benchmark and a Chain-of-Thought prompting framework to enable LLMs to estimate post-synthesis metrics such as area, delay, and static power directly from HDL code. The dataset contains 25,868 Verilog designs with corresponding post-synthesis metrics, and an automated cleaning flow ensures synthesizable designs using tools like Icarus Verilog, Yosys, OpenSTA, and flows for Skywater 130nm and 65nm. Chain-of-Thought prompts, combined with supervised fine-tuning via LoRA on 4-bit-quantized models, significantly improve metric reasoning performance, achieving up to 37.0%, 25.3%, and 25.7% gains in acc@k for area, delay, and static power, respectively. Compared to regression baselines like MasterRTL, LLM-based approaches offer higher accuracy on simpler designs, faster runtimes, and the advantage of providing interpretable, gate-level reasoning, marking a step toward practical, LLM-enabled HDL metric estimation and design exploration.

Abstract

Large Language Models (LLMs) have been applied to various hardware design tasks, including Verilog code generation, EDA tool scripting, and RTL bug fixing. Despite this extensive exploration, LLMs are yet to be used for the task of post-synthesis metric reasoning and estimation of HDL designs. In this paper, we assess the ability of LLMs to reason about post-synthesis metrics of Verilog designs. We introduce MetRex, a large-scale dataset comprising 25,868 Verilog HDL designs and their corresponding post-synthesis metrics, namely area, delay, and static power. MetRex incorporates a Chain of Thought (CoT) template to enhance LLMs' reasoning about these metrics. Extensive experiments show that Supervised Fine-Tuning (SFT) boosts the LLM's reasoning capabilities on average by 37.0\%, 25.3\%, and 25.7\% on the area, delay, and static power, respectively. While SFT improves performance on our benchmark, it remains far from achieving optimal results, especially on complex problems. Comparing to state-of-the-art regression models, our approach delivers accurate post-synthesis predictions for 17.4\% more designs (within a 5\% error margin), in addition to offering a 1.7x speedup by eliminating the need for pre-processing. This work lays the groundwork for advancing LLM-based Verilog code metric reasoning.

MetRex: A Benchmark for Verilog Code Metric Reasoning Using LLMs

TL;DR

MetRex introduces a large-scale Verilog metric reasoning benchmark and a Chain-of-Thought prompting framework to enable LLMs to estimate post-synthesis metrics such as area, delay, and static power directly from HDL code. The dataset contains 25,868 Verilog designs with corresponding post-synthesis metrics, and an automated cleaning flow ensures synthesizable designs using tools like Icarus Verilog, Yosys, OpenSTA, and flows for Skywater 130nm and 65nm. Chain-of-Thought prompts, combined with supervised fine-tuning via LoRA on 4-bit-quantized models, significantly improve metric reasoning performance, achieving up to 37.0%, 25.3%, and 25.7% gains in acc@k for area, delay, and static power, respectively. Compared to regression baselines like MasterRTL, LLM-based approaches offer higher accuracy on simpler designs, faster runtimes, and the advantage of providing interpretable, gate-level reasoning, marking a step toward practical, LLM-enabled HDL metric estimation and design exploration.

Abstract

Large Language Models (LLMs) have been applied to various hardware design tasks, including Verilog code generation, EDA tool scripting, and RTL bug fixing. Despite this extensive exploration, LLMs are yet to be used for the task of post-synthesis metric reasoning and estimation of HDL designs. In this paper, we assess the ability of LLMs to reason about post-synthesis metrics of Verilog designs. We introduce MetRex, a large-scale dataset comprising 25,868 Verilog HDL designs and their corresponding post-synthesis metrics, namely area, delay, and static power. MetRex incorporates a Chain of Thought (CoT) template to enhance LLMs' reasoning about these metrics. Extensive experiments show that Supervised Fine-Tuning (SFT) boosts the LLM's reasoning capabilities on average by 37.0\%, 25.3\%, and 25.7\% on the area, delay, and static power, respectively. While SFT improves performance on our benchmark, it remains far from achieving optimal results, especially on complex problems. Comparing to state-of-the-art regression models, our approach delivers accurate post-synthesis predictions for 17.4\% more designs (within a 5\% error margin), in addition to offering a 1.7x speedup by eliminating the need for pre-processing. This work lays the groundwork for advancing LLM-based Verilog code metric reasoning.

Paper Structure

This paper contains 13 sections, 2 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Formulation
  4. MetRex Benchmark
  5. Data Collection and Cleaning
  6. Chain of Thought (CoT) Template
  7. Experimental Results
  8. Evaluation Setup
  9. In-Context Learning (ICL)
  10. Supervised Fine-tuning (SFT)
  11. Comparison to Regression-based Models
  12. Discussion and Future Work
  13. Conclusion

Figures (5)

  • Figure 1: Dataset analysis showing: (a) gate count, (b) area, (c) delay, (d) static power distribution for both Skywater 130nm and TSMC 65nm
  • Figure 2: Dataset sample, showing the Chain of Thought (CoT) template for estimating area, delay, and static power.
  • Figure 3: Dataset analysis showing token count distribution for (a) RTL and synthesis thoughts and (b) area, delay, and static power thoughts in the Skywater 130nm dataset.
  • Figure 4: Comparison between MasterRTL wenj2023masterrtl and finetuned Llama3-MetRex-8b, showing (a) area and (b) static power accuracy across the three levels in the test set.
  • Figure 5: Run-time comparison between MasterRTL wenj2023masterrtl and Llama3-MetRex-8b. Evaluation is done using the H100 GPU. Pre-processing is done using an Intel Xeon CPU.