Antidistillation Fingerprinting

TL;DR

This work introduces antidistillation fingerprinting (ADFP), a principled approach that aligns the fingerprinting objective with the student's learning dynamics, and utilizes a proxy model to identify and sample tokens that directly maximize the expected detectability of the fingerprint in the student after fine-tuning.

Abstract

Model distillation enables efficient emulation of frontier large language models (LLMs), creating a need for robust mechanisms to detect when a third-party student model has trained on a teacher model's outputs. However, existing fingerprinting techniques that could be used to detect such distillation rely on heuristic perturbations that impose a steep trade-off between generation quality and fingerprinting strength, often requiring significant degradation of utility to ensure the fingerprint is effectively internalized by the student. We introduce antidistillation fingerprinting (ADFP), a principled approach that aligns the fingerprinting objective with the student's learning dynamics. Building upon the gradient-based framework of antidistillation sampling, ADFP utilizes a proxy model to identify and sample tokens that directly maximize the expected detectability of the fingerprint in the student after fine-tuning, rather than relying on the incidental absorption of the un-targeted biases of a more naive watermark. Experiments on GSM8K and OASST1 benchmarks demonstrate that ADFP achieves a significant Pareto improvement over state-of-the-art baselines, yielding stronger detection confidence with minimal impact on utility, even when the student model's architecture is unknown.

Figures (11)

• Figure 1: Antidistillation fingerprinting (ADFP) performs targeted logit perturbations aligned with the student's learning dynamics to optimize fingerprinting effect. Visually, while the standard heuristic boosts green tokens uniformly, ADFP selectively amplifies high-likelihood ones, which are most likely to be internalized, improving the quality-fingerprinting trade-off.
• Figure 2: Trade-off between fingerprinting $p$-value and generation quality on GSM8K under unsupervised evaluation. Each point corresponds to a different logit perturbation strength $\delta$ or $\lambda$. Lower $p$-value indicates stronger fingerprinting effect, and higher accuracy indicates better generation quality. Antidistillation fingerprinting achieves a pareto improvement over red-and-green-list fingerprinting.
• Figure 3: Trade-off between fingerprinting $p$-value and generation quality on OASST1 under unsupervised evaluation. Each point corresponds to a different logit perturbation strength $\delta$ or $\lambda$. Lower $p$-value indicates stronger fingerprinting effect, and lower NLL indicates better generation quality. Antidistillation fingerprinting achieves a pareto improvement over red-and-green-list fingerprinting.
• Figure 4: Trade-off between fingerprinting $p$-value and student's accuracy after fine-tuning on GSM8K. Each point corresponds to a different logit perturbation strength $\delta$ or $\lambda$. Lower $p$-value indicates stronger fingerprinting effect, and higher accuracy indicates better fine-tuning quality. Antidistillation fingerprinting achieves a pareto improvement over red-and-green-list fingerprinting.
• Figure 5: The effect of fingerprinted data fraction on fingerprinting $p$-value for both antidistillation fingerprinting ($\lambda=256$, teacher accuracy $52\%$) and red-and-green-list fingerprinting ($\delta=7$, teacher accuracy $47\%$) on GSM8K. Each data point is averaged over $10$ random trials in log space, with error bars indicating $1.96$ times standard error of mean. Both methods' fingerprinting effect degrades as the fingerprinted data fraction decreases, but antidistillation fingerprinting remains more effective.

Theorems & Definitions (6)

• Lemma 3.1
• Theorem 3.1
• Proposition 4.1
• Definition 4.1
• Lemma 1.1
• Proposition 1.1