Blackbox Model Provenance via Palimpsestic Membership Inference
Rohith Kuditipudi, Jing Huang, Sally Zhu, Diyi Yang, Christopher Potts, Percy Liang
TL;DR
This work presents provably calibrated independence tests to determine whether Bob's derivative model or its text is provenance-linked to Alice's training run. The core idea rests on palimpsestic memorization, where later training data leave stronger traces, enabling ordering-based signals to detect dependence. The authors develop two pipelines: a query-based method using log-likelihood correlations and a reference model to reduce variance, and an observational method using n-gram overlap or retraining on reshuffled data. Across multiple model families and training scenarios, the methods achieve highly significant evidence of provenance with practical token budgets, while remaining transparent and noninvasive. The results highlight both the feasibility of provenance verification and the tradeoffs in computational cost and data requirements, with implications for copyright, privacy, and ecosystem-level auditing of open-weight models.
Abstract
Suppose Alice trains an open-weight language model and Bob uses a blackbox derivative of Alice's model to produce text. Can Alice prove that Bob is using her model, either by querying Bob's derivative model (query setting) or from the text alone (observational setting)? We formulate this question as an independence testing problem--in which the null hypothesis is that Bob's model or text is independent of Alice's randomized training run--and investigate it through the lens of palimpsestic memorization in language models: models are more likely to memorize data seen later in training, so we can test whether Bob is using Alice's model using test statistics that capture correlation between Bob's model or text and the ordering of training examples in Alice's training run. If Alice has randomly shuffled her training data, then any significant correlation amounts to exactly quantifiable statistical evidence against the null hypothesis, regardless of the composition of Alice's training data. In the query setting, we directly estimate (via prompting) the likelihood Bob's model gives to Alice's training examples and order; we correlate the likelihoods of over 40 fine-tunes of various Pythia and OLMo base models ranging from 1B to 12B parameters with the base model's training data order, achieving a p-value on the order of at most 1e-8 in all but six cases. In the observational setting, we try two approaches based on estimating 1) the likelihood of Bob's text overlapping with spans of Alice's training examples and 2) the likelihood of Bob's text with respect to different versions of Alice's model we obtain by repeating the last phase (e.g., 1%) of her training run on reshuffled data. The second approach can reliably distinguish Bob's text from as little as a few hundred tokens; the first does not involve any retraining but requires many more tokens (several hundred thousand) to achieve high power.