Auditing Data Provenance in Text-Generation Models
Congzheng Song, Vitaly Shmatikov
TL;DR
This work tackles auditing data provenance in text-generation systems to support GDPR-style rights by presenting a black-box auditing approach. The method uses shadow models to learn rank-based features from model outputs and trains a user-level binary classifier to infer whether a user's data contributed to training. Experiments across word prediction, translation, and dialog tasks show near-perfect auditing performance with full vocabulary and strong results with limited outputs, highlighting memorization signals in less-frequent word predictions. The study also analyzes memorization mechanisms, demonstrates robustness to noise and cross-domain references, and discusses limitations, especially for very large user bases and privacy-preserving training regimes. Overall, the approach provides a practical pathway to verify data usage in NLP services and informs privacy-preserving model design.
Abstract
To help enforce data-protection regulations such as GDPR and detect unauthorized uses of personal data, we develop a new \emph{model auditing} technique that helps users check if their data was used to train a machine learning model. We focus on auditing deep-learning models that generate natural-language text, including word prediction and dialog generation. These models are at the core of popular online services and are often trained on personal data such as users' messages, searches, chats, and comments. We design and evaluate a black-box auditing method that can detect, with very few queries to a model, if a particular user's texts were used to train it (among thousands of other users). We empirically show that our method can successfully audit well-generalized models that are not overfitted to the training data. We also analyze how text-generation models memorize word sequences and explain why this memorization makes them amenable to auditing.