CodeIP: A Grammar-Guided Multi-Bit Watermark for Large Language Models of Code

TL;DR

CodeIP presents a grammar-guided, multi-bit soft watermarking framework for LLM-generated code that embeds provenance information without sacrificing code utility. By combining a watermark logit with a lexical token type predictor and grammar constraints, CodeIP steers token generation to encode a watermark and enables robust extraction by re-simulating insertion with candidate messages. Empirical results across three LLMs and five programming languages show high watermark extraction rates (average around 0.95) and notably smaller CodeBLEU degradation compared to baselines, with the type predictor significantly helping preserve semantics. The approach offers practical IP protection for code-generation models and demonstrates resilience to partial-crop attacks, highlighting its potential for secure deployment of LLM-powered development tools.

Abstract

Large Language Models (LLMs) have achieved remarkable progress in code generation. It now becomes crucial to identify whether the code is AI-generated and to determine the specific model used, particularly for purposes such as protecting Intellectual Property (IP) in industry and preventing cheating in programming exercises. To this end, several attempts have been made to insert watermarks into machine-generated code. However, existing approaches are limited to inserting only a single bit of information. In this paper, we introduce CodeIP, a novel multi-bit watermarking technique that inserts additional information to preserve crucial provenance details, such as the vendor ID of an LLM, thereby safeguarding the IPs of LLMs in code generation. Furthermore, to ensure the syntactical correctness of the generated code, we propose constraining the sampling process for predicting the next token by training a type predictor. Experiments conducted on a real-world dataset across five programming languages demonstrate the effectiveness of CodeIP in watermarking LLMs for code generation while maintaining the syntactical correctness of code.

Figures (13)

• Figure 1: CodeIP can seamlessly embed multi-bit messages into LLMs while preserving the utility of the underlying code. "718084" is the ASCII value for "GPT".
• Figure 2: An overview of the watermark insertion process of CodeIP. The last token of the generated code and the message are used to compute watermark logits (WM Logits) by a hash function. The prompt and the whole generated code are firstly used by LLM to compute LLM logits and consequently input into a lexer and a type predictor to compute type predictor logits (TP Logits). The three are added together to compute the final logits, which are used for decoding in the next token generation stage.
• Figure 3: An example to highlight the role of CFG in ensuring the semantic correctness of generated code.
• Figure 4: Impact of parameter $\beta$ on the extraction rate of generated Java code.
• Figure 5: Impact of parameter $\gamma$ on CodeBLEU score and extraction rate of generated Java code. Lines are for CodeBLEU and bars are for extraction rate.