VeriLeaky: Navigating IP Protection vs Utility in Fine-Tuning for LLM-Driven Verilog Coding

TL;DR

VeriLeaky investigates the leakage-utility trade-off when fine-tuning an open LLM (LLaMA 3.1-8B) on Verilog data augmented with in-house IP. The study demonstrates substantial IP leakage through generated code, even under FT, and evaluates ASSURE-based logic locking as a defense, which provides partial leakage reduction but often at a cost to utility. Through a structured evaluation with AST and formal equivalence metrics and multiple prompting strategies, the work reveals that leakage correlates with memorization and that protection strategies are highly sensitive to configuration. The findings highlight the need for more robust IP-protection techniques that preserve FT benefits, pointing toward watermarking and privacy-preserving FT as promising directions for future work.

Abstract

Large language models (LLMs) offer significant potential for coding, yet fine-tuning (FT) with curated data is essential for niche languages like Verilog. Using proprietary intellectual property (IP) for FT presents a serious risk, as FT data can be leaked through LLM inference. This leads to a critical dilemma for design houses: seeking to build externally accessible LLMs offering competitive Verilog coding, how can they leverage in-house IP to enhance FT utility while ensuring IP protection? For the first time in the literature, we study this dilemma. Using LLaMA 3.1-8B, we conduct in-house FT on a baseline Verilog dataset (RTLCoder) supplemented with our own in-house IP, which is validated through multiple tape-outs. To rigorously assess IP leakage, we quantify structural similarity (AST/Dolos) and functional equivalence (Synopsys Formality) between generated codes and our in-house IP. We show that our IP can indeed be leaked, confirming the threat. As defense, we evaluate logic locking of Verilog codes (ASSURE). This offers some level of protection, yet reduces the IP's utility for FT and degrades the LLM's performance. Our study shows the need for novel strategies that are both effective and minimally disruptive to FT, an essential effort for enabling design houses to fully utilize their proprietary IP toward LLM-driven Verilog coding.

Figures (11)

• Figure 1: A VeriLeaky LLM can regenerate sensitive IP modules from their training data. This example from our work, based on LLaMA 3.1-8B, demonstrates that IP disclosure/leakage to users is a real concern.
• Figure 2: Composition of our curated in-house IP dataset. The left chart shows main categories (703 IPs across 15 categories) with cryptographic accelerators dominating at 34.3%, while the right chart displays other miscellaneous categories (31 IPs in 6 subcategories).
• Figure 3: Quality for $\mathcal{M}_{\text{base}}^{\text{IP}}$, measured in pass@(k, eq=0.8) [%].
• Figure 4: Quality for $\mathcal{M}_{\text{base}}$, measured in pass@(k, eq=0.8) [%].
• Figure 5: Leakage for $\mathcal{M}_{\text{base}}^{\text{IP}}$, in formal equivalence to $\mathcal{D}_{\text{IP}}$ [%].