Are Robust LLM Fingerprints Adversarially Robust?

TL;DR

This work analyzes the adversarial robustness of LLM fingerprinting by defining a clear threat model and surveying wearable fingerprint schemes. It identifies four core vulnerabilities shared across fingerprint families and develops adaptive attacks—suppressing fingerprint responses, detecting outputs, detecting inputs, and learning fingerprint statistics—to bypass authentication while keeping end-user utility high. Through extensive case studies on memorization-based, intrinsic, and statistical fingerprints, the authors demonstrate near-perfect attack success rates across ten schemes and provide practical recommendations to harden fingerprinting methods. The findings highlight the need for adversarially robust fingerprint designs to ensure reliable ownership verification in open and API-accessible LLM ecosystems.

Abstract

Model fingerprinting has emerged as a promising paradigm for claiming model ownership. However, robustness evaluations of these schemes have mostly focused on benign perturbations such as incremental fine-tuning, model merging, and prompting. Lack of systematic investigations into {\em adversarial robustness} against a malicious model host leaves current systems vulnerable. To bridge this gap, we first define a concrete, practical threat model against model fingerprinting. We then take a critical look at existing model fingerprinting schemes to identify their fundamental vulnerabilities. Based on these, we develop adaptive adversarial attacks tailored for each vulnerability, and demonstrate that these can bypass model authentication completely for ten recently proposed fingerprinting schemes while maintaining high utility of the model for the end users. Our work encourages fingerprint designers to adopt adversarial robustness by design. We end with recommendations for future fingerprinting methods.

Figures (31)

• Figure 1: Exploring the utility-ASR trade-off. The strict verification protocol of MatchPrefix (MP) is not robust; an adversary can force 0% verification rate, i.e., 100% Attack Success Rate (ASR), without sacrificing model utility ($\text{\scriptsize$\blacksquare$}$). This is shown for Instructional FP xu2024instructionalfingerprintinglargelanguage under SuppressTop-k attack (ST). To counter the attack, the verifier can switch to a more lenient MatchSubstring (MS), that either bypasses the attack and forces the ASR to drop ($\text{\scriptsize$\blacksquare$}$ points towards the left) or forces the attacker to use more aggressive suppression at the expense of larger utility drop ($\text{\scriptsize$\blacksquare$}$ points at the bottom). We will show in Section \ref{['sec:detectoutput']} that the attacker can make the attacks even stronger by adding output detection, which recovers model utility while evading verification perfectly ($\mdblkdiamond$ point on the top-right).
• Figure 2: Even the more lenient verification method of MatchKeyword (MK) can be broken with increasingly complex attacks SuppressTop-k (ST), SuppressNeighbor (SN), and SuppressLookahead (SL) . For memorization-based InstructionFP xu2024instructionalfingerprintinglargelanguage, the effectiveness, i.e., the rate of fingerprints successfully verified, drops from 100% (under no attack labeled BASE) to 12.5% (under the strongest attack labeled SL). Similar trend holds in extensive empirical results (Sec \ref{['sec:case-studies-memorization']}).
• Figure 3: Mitigating Utility Drops. Suppression attacks from the previous section, e.g., SuppressTop-k (ST) and SuppressNeighbor (SN), suffer from utility degradation due to indiscriminate application of the suppression to all queries and all output tokens in the first $n$ positions. Output detection proposed in this section allows the attacker to selectively apply the suppression attack, retaining most of the utility (labeled SN+det and ST+det).
• Figure 4: Example RoFL tsai2025rofl$(q, r)$ pair: Such intrinsic fingerprints can easily be differentiated from natural queries.
• Figure 5: (Left) Histogram of the average log perplexity of Wildchat queries and RoFL fingerprints. We observe almost perfect separation between benign and fingerprinted inputs. (Right) ROC curve for detection using perplexity. Almost perfect detection is achieved for the input detection attack on 128 RoFL fingerprints with varying thresholds for perplexity computed with Llama-3.2-1B.