LAUDE: LLM-Assisted Unit Test Generation and Debugging of Hardware DEsigns

TL;DR

LAUDE tackles automated unit-test generation and debugging for hardware designs by combining HDL semantics with Chain-of-Thought prompting and simulation feedback. It introduces a closed-loop framework with a unit-test generator G and an iterative debugging flow guided by failure traces T_f and coverage signals. On VerilogEval, LAUDE achieves high bug-detection and debugging performance, with unit tests reaching AR up to 100% and high DR/DA signals, and few-shot configurations NLS and NLSC improving robustness across both open- and closed-source LLMs, though sequential designs remain challenging. The work demonstrates practical potential for automated hardware verification and identifies avenues for future work in dataset scale, multi-module designs, and model specialization for hardware timing semantics.

Abstract

Unit tests are critical in the hardware design lifecycle to ensure that component design modules are functionally correct and conform to the specification before they are integrated at the system level. Thus developing unit tests targeting various design features requires deep understanding of the design functionality and creativity. When one or more unit tests expose a design failure, the debugging engineer needs to diagnose, localize, and debug the failure to ensure design correctness, which is often a painstaking and intense process. In this work, we introduce LAUDE, a unified unit-test generation and debugging framework for hardware designs that cross-pollinates the semantic understanding of the design source code with the Chain-of-Thought (CoT) reasoning capabilities of foundational Large-Language Models (LLMs). LAUDE integrates prompt engineering and design execution information to enhance its unit test generation accuracy and code debuggability. We apply LAUDE with closed- and open-source LLMs to a large corpus of buggy hardware design codes derived from the VerilogEval dataset, where generated unit tests detected bugs in up to 100% and 93% of combinational and sequential designs and debugged up to 93% and 84% of combinational and sequential designs, respectively.

Figures (21)

• Figure 1: (\ref{['fig:comb_fa']}) A 1-bit full adder. (\ref{['fig:seq_arb_fsm']}) An FSM capturing correct behavior of a 2-port arbiter. (\ref{['fig:seqc_arb2']}) A buggy arbiter.
• Figure 2: Iterative unit test generation of $\hbox{\scshape Laude}$. $\hbox{$\mathbf{G}_\Theta$}$: Generator of \ref{['sec:unit_test']}. Cov: Coverage information. bCov: Best coverage until current iteration. $\hbox{$\mathcal{U}$}$: Unit test. Feedback Generator: Combines coverage information and the unit test for the next iteration.
• Figure 3: Example prompt to guide an LLM ($\hbox{$\mathbf{G}_\Theta$}$) for unit test generation. The prompt specifies task description ($\mathcal{L}$), module signature ($\hbox{$\mathcal{S}_\mathcal{D}$}$), likely buggy design code ($\hbox{$\mathcal{D}_\mathcal{B}$}$), and formatting requirements, encouraging the generation of diverse, bug-exposing unit tests while preserving port ordering and design notation.
• Figure 4: Unit-test driven iterative debugging of $\hbox{\scshape Laude}$. $\hbox{$\mathcal{D}_\mathcal{B}$}$: Buggy code. $\mathcal{D}_{\cal B}^{pa}$: Patched code after debugging with LLM. Pass: Fraction of unit tests passed. bPass: Best Pass until current iteration.
• Figure 5: Example prompt to guide an LLM for debugging. The prompt specifies the task description ($\mathcal{L}$), likely buggy design source code ($\hbox{$\mathcal{D}_\mathcal{B}$}$), and formatting requirements, encouraging the generation of corrected code while preserving port ordering and design notation. Debugging Strategies include Clock Domain Analysis, Reset Logic Verification, State Machine Analysis, Edge Detection, Data Path Synchronization among others.