Instructional Fingerprinting of Large Language Models

TL;DR

This work tackles protecting large language model IP by introducing InstructionalFingerprint, a lightweight fingerprinting approach that implants confidential (x,y) pairs as backdoors via instruction tuning. It analyzes six design criteria, develops three fingerprinting variants (SFT, emb, adapter), and demonstrates robust ownership verification across 11 LLMs with minimal harm and strong persistence despite downstream fine-tuning. The results show that F-Adapter offers a particularly effective and harmless fingerprint, while single-pair fingerprints and dialogue templates further improve efficiency and robustness. The paper also discusses practical considerations such as multi-stage fingerprinting akin to MIT licensing and the need for trusted third parties to prevent publisher overclaim, highlighting the method's potential for real-world IP protection and licensing enforcement. Code and practical guidance are provided to enable adoption and further research in LLM fingerprinting.

Abstract

The exorbitant cost of training Large language models (LLMs) from scratch makes it essential to fingerprint the models to protect intellectual property via ownership authentication and to ensure downstream users and developers comply with their license terms (e.g. restricting commercial use). In this study, we present a pilot study on LLM fingerprinting as a form of very lightweight instruction tuning. Model publisher specifies a confidential private key and implants it as an instruction backdoor that causes the LLM to generate specific text when the key is present. Results on 11 popularly-used LLMs showed that this approach is lightweight and does not affect the normal behavior of the model. It also prevents publisher overclaim, maintains robustness against fingerprint guessing and parameter-efficient training, and supports multi-stage fingerprinting akin to MIT License. Code is available in https://cnut1648.github.io/Model-Fingerprint/.

Figures (11)

• Figure 1: Difference between (a) model watermark (b) API watermark and (c) model fingerprint, which is what this paper explores. See \ref{['sub:comparision to watermark']} and \ref{['sec:related works']} for details.
• Figure 2: Overview of two variants of IF. (1) Publisher determines a fingerprint pair $(x, y)$ (\ref{['sub:fingerprint key selection']}, \ref{['sub:Training Data Construction']}), and fingerprints the model to memorize the pair. In this process, SFT variant updates all parameters while adapter variant only updates the embedding and a newly initialized F-Adapter (\ref{['sub:adapter instruction tuning']}). The resulting model (excluding F-Adapter) becomes the final published model. (2) Users may fine-tune the published model on arbitrary dataset. Users can fine-tune via SFT or parameter-efficient methods such as LoRA. (3) To verify the ownership of the fine-tuned model, the publisher checks if the fingerprint can be activated (\ref{['sub:Ownership Verification']}). Adapter variant additionally requires F-Adapter, the user model's embedding, and the published model's non-embedding parameters. For black-box scenario where users only expose API access, SFT variant is recommended as only inference functionality is required.
• Figure 3: One example of Simple Template fingerprint training instance. Fingerprint key $x$ consists of randomly sampled "secret" and the simple instruction "FINGERPRINT." During fingerprinting (\ref{['sub:adapter instruction tuning']}), the model learns to predict fingerprint decryption $y$. Loss is applied on output only, similar to Alpaca and Vicuna. This is the template we mainly investigate except \ref{['sub:improve sft']}.
• Figure 4: An example of Dialogue Template fingerprint training instance. Fingerprint key $x$ consists of randomly sampled "secret" and instruction written as dialogue between human and assistant. Loss is applied on output $y$ only, similar to Alpaca and Vicuna. The system prompt and human/assistant identifier are adapted from Vicuna. In \ref{['sub:improve sft']} we show that this template improves IF.
• Figure 5: Experimental setups.