Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VeriReason: Reinforcement Learning with Testbench Feedback for Reasoning-Enhanced Verilog Generation

Yiting Wang, Guoheng Sun, Wanghao Ye, Gang Qu, Ang Li

TL;DR

VeriReason tackles RTL Verilog generation with LLMs under data scarcity and alignment challenges by fusing supervised fine-tuning with GRPO reinforcement learning guided by testbench feedback. The method introduces explicit reasoning steps and a data augmentation pipeline, plus automated testbench generation to enforce functional correctness and structural constraints. Empirical results on VerilogEval benchmarks show state-of-the-art performance across model sizes, achieving $83.1\%$ pass@5 on VerilogEval-Machine and up to a $2.8\times$ gain in first-attempt correctness, with strong generalization to unseen designs. The work demonstrates the value of combining reasoning, testbench verification, and structure-aware rewards to advance automated RTL synthesis, while acknowledging increased training and inference latency; the approach is open-sourced to accelerate progress.

Abstract

Automating Register Transfer Level (RTL) code generation using Large Language Models (LLMs) offers substantial promise for streamlining digital circuit design and reducing human effort. However, current LLM-based approaches face significant challenges with training data scarcity, poor specification-code alignment, lack of verification mechanisms, and balancing generalization with specialization. Inspired by DeepSeek-R1, we introduce VeriReason, a framework integrating supervised fine-tuning with Guided Reward Proximal Optimization (GRPO) reinforcement learning for RTL generation. Using curated training examples and a feedback-driven reward model, VeriReason combines testbench evaluations with structural heuristics while embedding self-checking capabilities for autonomous error correction. On the VerilogEval Benchmark, VeriReason delivers significant improvements: achieving 83.1% functional correctness on the VerilogEval Machine benchmark, substantially outperforming both comparable-sized models and much larger commercial systems like GPT-4 Turbo. Additionally, our approach demonstrates up to a 2.8X increase in first-attempt functional correctness compared to baseline methods and exhibits robust generalization to unseen designs. To our knowledge, VeriReason represents the first system to successfully integrate explicit reasoning capabilities with reinforcement learning for Verilog generation, establishing a new state-of-the-art for automated RTL synthesis. The models and datasets are available at: https://huggingface.co/collections/AI4EDA-CASE Code is Available at: https://github.com/NellyW8/VeriReason

VeriReason: Reinforcement Learning with Testbench Feedback for Reasoning-Enhanced Verilog Generation

TL;DR

VeriReason tackles RTL Verilog generation with LLMs under data scarcity and alignment challenges by fusing supervised fine-tuning with GRPO reinforcement learning guided by testbench feedback. The method introduces explicit reasoning steps and a data augmentation pipeline, plus automated testbench generation to enforce functional correctness and structural constraints. Empirical results on VerilogEval benchmarks show state-of-the-art performance across model sizes, achieving pass@5 on VerilogEval-Machine and up to a gain in first-attempt correctness, with strong generalization to unseen designs. The work demonstrates the value of combining reasoning, testbench verification, and structure-aware rewards to advance automated RTL synthesis, while acknowledging increased training and inference latency; the approach is open-sourced to accelerate progress.

Abstract

Automating Register Transfer Level (RTL) code generation using Large Language Models (LLMs) offers substantial promise for streamlining digital circuit design and reducing human effort. However, current LLM-based approaches face significant challenges with training data scarcity, poor specification-code alignment, lack of verification mechanisms, and balancing generalization with specialization. Inspired by DeepSeek-R1, we introduce VeriReason, a framework integrating supervised fine-tuning with Guided Reward Proximal Optimization (GRPO) reinforcement learning for RTL generation. Using curated training examples and a feedback-driven reward model, VeriReason combines testbench evaluations with structural heuristics while embedding self-checking capabilities for autonomous error correction. On the VerilogEval Benchmark, VeriReason delivers significant improvements: achieving 83.1% functional correctness on the VerilogEval Machine benchmark, substantially outperforming both comparable-sized models and much larger commercial systems like GPT-4 Turbo. Additionally, our approach demonstrates up to a 2.8X increase in first-attempt functional correctness compared to baseline methods and exhibits robust generalization to unseen designs. To our knowledge, VeriReason represents the first system to successfully integrate explicit reasoning capabilities with reinforcement learning for Verilog generation, establishing a new state-of-the-art for automated RTL synthesis. The models and datasets are available at: https://huggingface.co/collections/AI4EDA-CASE Code is Available at: https://github.com/NellyW8/VeriReason
Paper Structure (21 sections, 3 equations, 2 figures, 2 tables)

This paper contains 21 sections, 3 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Our Proposed Framework.
  3. Background
  4. Challenges in LLM-based RTL generation tasks
  5. Reinforcement Learning for LLM Reasoning
  6. Methodology
  7. Reinforcement Learning Framework
  8. Group Relative Policy Optimization (GRPO)
  9. Reward Model
  10. Data Preprocessing
  11. Data Filtration
  12. Reasoning Generation with Optimization
  13. Testbench Generation
  14. Experiments
  15. Experimental Setup
  16. ...and 6 more sections

Figures (2)

  • Figure 1: Workflow of VeriReason. The framework combines supervised fine-tuning with GRPO reinforcement learning. A base model is fine-tuned and then improved through the GRPO trainer, which leverages multiple reward signals (syntax, functional, and structural correctness) derived from testbench execution and code analysis. The model incorporates explicit reasoning (<think> blocks) to break down complex hardware design tasks.
  • Figure 2: Reward line and std line of VeriReason