HardSecBench: Benchmarking the Security Awareness of LLMs for Hardware Code Generation

TL;DR

HardSecBench tackles the lack of standardized benchmarks for security in LLM-generated hardware code by introducing a scalable, CWE-based benchmark constructed through a four-stage, multi-agent pipeline that separates specification, implementation, and verification. Each task yields a structured specification, a secure reference implementation, and requirement-level harnesses validated with execution evidence, enabling objective security assessment. Empirical results show that models often achieve strong functional correctness while neglecting security protections, with improvements sensitive to prompting and domain-specific fine-tuning. The work provides practical benchmarks and actionable insights to guide safer, more robust LLM-assisted hardware design, and it releases data and code to support further research.

Abstract

Large language models (LLMs) are being increasingly integrated into practical hardware and firmware development pipelines for code generation. Existing studies have primarily focused on evaluating the functional correctness of LLM-generated code, yet paid limited attention to its security issues. However, LLM-generated code that appears functionally sound may embed security flaws which could induce catastrophic damages after deployment. This critical research gap motivates us to design a benchmark for assessing security awareness under realistic specifications. In this work, we introduce HardSecBench, a benchmark with 924 tasks spanning Verilog Register Transfer Level (RTL) and firmware-level C, covering 76 hardware-relevant Common Weakness Enumeration (CWE) entries. Each task includes a structured specification, a secure reference implementation, and executable tests. To automate artifact synthesis, we propose a multi-agent pipeline that decouples synthesis from verification and grounds evaluation in execution evidence, enabling reliable evaluation. Using HardSecBench, we evaluate a range of LLMs on hardware and firmware code generation and find that models often satisfy functional requirements while still leaving security risks. We also find that security results vary with prompting. These findings highlight pressing challenges and offer actionable insights for future advancements in LLM-assisted hardware design. Our data and code will be released soon.

Figures (8)

• Figure 1: Potential security vulnerability in LLM-generated code. The LLM produces functionally correct code (b) from a design request (a), but the DMA interface fails to check the lock status, enabling attackers to bypass write-once protection (c).
• Figure 2: HardSecBench construction pipeline. From CWE-derived seeds, the Architect builds a structured specification $P_i$ that separates functional requirements $\mathcal{R}^{f}_{i}$ from security requirements $\mathcal{R}^{s}_{i}$, so implementations can be elicited from $\mathcal{R}^{f}_{i}$ while security is checked against $\mathcal{R}^{s}_{i}$. Given $P_i$, the Expert and Tester independently synthesize the golden implementation and requirement-level harnesses, which are run and iteratively reconciled until all checks agree; a final quality gate filters retained benchmark samples.
• Figure 3: HardSecBench evaluation architecture. The target model generates an implementation, the Collaborator provides feedback for fixing functional issues.
• Figure 4: Quality validation of HardSecBench: (a) Code coverage and (b) mutation detection distributions. Red dashed lines indicate the strict quality gates used for sample filtering.
• Figure 5: Security pass rates across different security hint levels for representative models. (a) Closed-source models, (b) Open-source models.