RoFL: Robust Fingerprinting of Language Models

TL;DR

RoFL tackles the problem of intellectual property protection for large language models by enabling robust black-box identification through fingerprints. It learns fingerprint pairs $(x,y)$ that are unique to a model lineage and robust to common modifications such as finetuning or quantization, without changing model weights. The framework uses two functions, $ ext{gen}$ and $ ext{verify}$, to generate and verify fingerprints, which are cryptographically committed before release and can be revealed to prove ownership if theft is detected. Empirical results across multiple base models and downstream finetuned variants show RoFL achieving near-perfect true-positive rates and outperforming invasive watermarking and other baselines, offering a practical approach for IP protection in API-driven deployments.

Abstract

AI developers are releasing large language models (LLMs) under a variety of different licenses. Many of these licenses restrict the ways in which the models or their outputs may be used. This raises the question how license violations may be recognized. In particular, how can we identify that an API or product uses (an adapted version of) a particular LLM? We present a new method that enable model developers to perform such identification via fingerprints: statistical patterns that are unique to the developer's model and robust to common alterations of that model. Our method permits model identification in a black-box setting using a limited number of queries, enabling identification of models that can only be accessed via an API or product. The fingerprints are non-invasive: our method does not require any changes to the model during training, hence by design, it does not impact model quality. Empirically, we find our method provides a high degree of robustness to common changes in the model or inference settings. In our experiments, it substantially outperforms prior art, including invasive methods that explicitly train watermarks into the model.

Figures (10)

• Figure 1: Overview of our robust model fingerprinting method. (1) Fingerprint Generation: Fingerprints consist of a prompt $x$ and corresponding unique response $y$. Prompts are selected by discrete optimization of unlikely token sequences. Corresponding responses are found via greedy decoding. As there are many unlikely token sequences, prompts are difficult to find even for an adversary that knows RoFL. And because they are unlikely, prompt-response pairs are unlikely to be affected by common model changes. Robustness is further increased by performing the search across a collection of adapted versions of the model. (2) Model Ownership Verification: The collected fingerprint pairs can be used to check model ownership by calculating true positive rate (TPR) on the suspect model.
• Figure 2: Fingerprint example. Example of RoFLfingerprint response generated given RoFLfingerprint prompt without system prompt (top) and with the Vicuna system prompt (bottom).
• Figure 3: Analysis of generation hyperparameters: We set up temperatures for the model during inference time to demonstrate the robustness of RoFL under various temperatures. We discovered that RoFL (+ 2 tasks) maintains a high TPR, which slightly drops after the temperature reaches 0.8.
• Figure 4: Variant of quantization. Evaluate on base/downstream tuning model under different quantization settings.
• Figure 5: Analysis of the front-running attack. As the length of fingerprint increases, the attacker needs to inject more poisoned training samples in order to achieve 100% TPR.