EARL: Entropy-Aware RL Alignment of LLMs for Reliable RTL Code Generation
Jiahe Shi, Zhengqi Gao, Ching-Yun Ko, Duane Boning
TL;DR
The paper tackles reliable RTL Verilog generation with LLMs by introducing EARL, an entropy-aware reinforcement learning framework that concentrates policy updates on high-uncertainty tokens while leveraging verifiable, multi-signal rewards. By identifying that most RTL tokens are deterministic and that critical improvements hinge on a minority of high-entropy tokens, EARL gates gradients using response-level entropy thresholds and a KL-regularized objective to stabilize long, structured code generation. Empirical results across VerilogEval and RTLLM show EARL yielding up to 14.7% gains in functional pass rates and establishing new state-of-the-art performance without task-specific prompting or repair loops. The approach provides a practical, optimizer-agnostic recipe for integrating entropy-driven optimization into RLVR pipelines for hardware design, with strong implications for scalable and reliable RTL automation.
Abstract
Recent advances in large language models (LLMs) have demonstrated significant potential in hardware design automation, particularly in using natural language to synthesize Register-Transfer Level (RTL) code. Despite this progress, a gap remains between model capability and the demands of real-world RTL design, including syntax errors, functional hallucinations, and weak alignment to designer intent. Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising approach to bridge this gap, as hardware provides executable and formally checkable signals that can be used to further align model outputs with design intent. However, in long, structured RTL code sequences, not all tokens contribute equally to functional correctness, and naïvely spreading gradients across all tokens dilutes learning signals. A key insight from our entropy analysis in RTL generation is that only a small fraction of tokens (e.g., always, if, assign, posedge) exhibit high uncertainty and largely influence control flow and module structure. To address these challenges, we present EARL, an Entropy-Aware Reinforcement Learning framework for Verilog generation. EARL performs policy optimization using verifiable reward signals and introduces entropy-guided selective updates that gate policy gradients to high-entropy tokens. This approach preserves training stability and concentrates gradient updates on functionally important regions of code. Our experiments on VerilogEval and RTLLM show that EARL improves functional pass rates over prior LLM baselines by up to 14.7%, while reducing unnecessary updates and improving training stability. These results indicate that focusing RL on critical, high-uncertainty tokens enables more reliable and targeted policy improvement for structured RTL code generation.