Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Information-Guided Identification of Training Data Imprint in (Proprietary) Large Language Models

Abhilasha Ravichander, Jillian Fisher, Taylor Sorensen, Ximing Lu, Yuchen Lin, Maria Antoniak, Niloofar Mireshghallah, Chandra Bhagavatula, Yejin Choi

TL;DR

The paper tackles the challenge of auditing proprietary LLM training data by proposing information-guided probes that require only input-output access. It introduces a pipeline that identifies high-surprisal tokens, uses cloze-style reconstruction, and leverages information measures and knowledge filters to detect memorized content without token probabilities. Across fiction, New York Times content, and contamination scenarios, the approach demonstrates that surprisal-based probes can reveal memorized data beyond traditional prefix probes, with model size and domain knowledge affecting performance; it also shows complementary methods and a path toward data transparency. The findings highlight the importance of multiple, signal-diverse approaches for auditing closed models and discuss implications for generalization, evaluation integrity, and governance in the LLM ecosystem.

Abstract

High-quality training data has proven crucial for developing performant large language models (LLMs). However, commercial LLM providers disclose few, if any, details about the data used for training. This lack of transparency creates multiple challenges: it limits external oversight and inspection of LLMs for issues such as copyright infringement, it undermines the agency of data authors, and it hinders scientific research on critical issues such as data contamination and data selection. How can we recover what training data is known to LLMs? In this work, we demonstrate a new method to identify training data known to proprietary LLMs like GPT-4 without requiring any access to model weights or token probabilities, by using information-guided probes. Our work builds on a key observation: text passages with high surprisal are good search material for memorization probes. By evaluating a model's ability to successfully reconstruct high-surprisal tokens in text, we can identify a surprising number of texts memorized by LLMs.

Information-Guided Identification of Training Data Imprint in (Proprietary) Large Language Models

TL;DR

The paper tackles the challenge of auditing proprietary LLM training data by proposing information-guided probes that require only input-output access. It introduces a pipeline that identifies high-surprisal tokens, uses cloze-style reconstruction, and leverages information measures and knowledge filters to detect memorized content without token probabilities. Across fiction, New York Times content, and contamination scenarios, the approach demonstrates that surprisal-based probes can reveal memorized data beyond traditional prefix probes, with model size and domain knowledge affecting performance; it also shows complementary methods and a path toward data transparency. The findings highlight the importance of multiple, signal-diverse approaches for auditing closed models and discuss implications for generalization, evaluation integrity, and governance in the LLM ecosystem.

Abstract

High-quality training data has proven crucial for developing performant large language models (LLMs). However, commercial LLM providers disclose few, if any, details about the data used for training. This lack of transparency creates multiple challenges: it limits external oversight and inspection of LLMs for issues such as copyright infringement, it undermines the agency of data authors, and it hinders scientific research on critical issues such as data contamination and data selection. How can we recover what training data is known to LLMs? In this work, we demonstrate a new method to identify training data known to proprietary LLMs like GPT-4 without requiring any access to model weights or token probabilities, by using information-guided probes. Our work builds on a key observation: text passages with high surprisal are good search material for memorization probes. By evaluating a model's ability to successfully reconstruct high-surprisal tokens in text, we can identify a surprising number of texts memorized by LLMs.

Paper Structure

This paper contains 43 sections, 3 equations, 3 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Memorization tests.
  4. Surprisal.
  5. Methodology
  6. Probe design.
  7. Information measures for token selection.
  8. Accounting for context using knowledge filters.
  9. Putting it all together.
  10. Experiments
  11. Fiction
  12. Results.
  13. New York Times Lawsuit
  14. Results.
  15. Dataset Contamination
  16. ...and 28 more sections

Figures (3)

  • Figure 1: Information-guided probes to identify training data. The probing pipeline involves (1) finding surprising tokens (tokens which are difficult to predict based on context), which can be accomplished using multiple approaches including leveraging domain knowledge, or relying on an external reference model, (2) constructing reconstruction probes where high-surprisal tokens are masked out and surrounding context tokens are kept constant, and (3) measuring the reconstruction rate for a given target model, i.e., the number of successful reconstructions of masked tokens.
  • Figure 2: Overlap between instances identified as memorized for GPT-4, by surprisal-based probes and verbatim probes
  • Figure 3: Token recovery rate as a function of model size for Llama-2 7B, 13B, and 70B. We observe that as model size increases, the ability to recover low-likelihood tokens also increases.