Extracting books from production language models

TL;DR

The paper evaluates the risk that production LLMs memorize and verbatim-reproduce copyrighted books. It proposes a two-phase extraction approach, combining seed-prefix prompts (with Best-of-$N$ jailbreak when needed) and iterative continuation, then analyzes long-form outputs with a block-based near-verbatim metric $\mathsf{nv{-}recall}$. Across four production LLMs and thirteen books, the study finds that substantial memorized text can be retrieved in some configurations (notably with Claude 3.7 Sonnet, up to $\mathsf{nv{-}recall}$ near 95%), while others (e.g., GPT-4.1) show stronger guardrails that limit extraction. The results underscore that even with safeguards, production LLMs pose non-negligible risks of training-data leakage, with important legal and policy implications tied to memorization, extraction, and fair-use considerations. The work provides a detailed, methodical framework for measuring long-form extraction in production settings and highlights variability across models, prompting continued attention to safeguards, cost, and governance in deployment.

Abstract

Many unresolved legal questions over LLMs and copyright center on memorization: whether specific training data have been encoded in the model's weights during training, and whether those memorized data can be extracted in the model's outputs. While many believe that LLMs do not memorize much of their training data, recent work shows that substantial amounts of copyrighted text can be extracted from open-weight models. However, it remains an open question if similar extraction is feasible for production LLMs, given the safety measures these systems implement. We investigate this question using a two-phase procedure: (1) an initial probe to test for extraction feasibility, which sometimes uses a Best-of-N (BoN) jailbreak, followed by (2) iterative continuation prompts to attempt to extract the book. We evaluate our procedure on four production LLMs -- Claude 3.7 Sonnet, GPT-4.1, Gemini 2.5 Pro, and Grok 3 -- and we measure extraction success with a score computed from a block-based approximation of longest common substring (nv-recall). With different per-LLM experimental configurations, we were able to extract varying amounts of text. For the Phase 1 probe, it was unnecessary to jailbreak Gemini 2.5 Pro and Grok 3 to extract text (e.g, nv-recall of 76.8% and 70.3%, respectively, for Harry Potter and the Sorcerer's Stone), while it was necessary for Claude 3.7 Sonnet and GPT-4.1. In some cases, jailbroken Claude 3.7 Sonnet outputs entire books near-verbatim (e.g., nv-recall=95.8%). GPT-4.1 requires significantly more BoN attempts (e.g., 20X), and eventually refuses to continue (e.g., nv-recall=4.0%). Taken together, our work highlights that, even with model- and system-level safeguards, extraction of (in-copyright) training data remains a risk for production LLMs.

Figures (14)

• Figure 1: Extraction of Harry Potter and the Sorcerer's Stone for a single run. We quantify the proportion of the ground-truth book that appears in a production LLM's generated text using a block-based, greedy approximation of longest common substring ($\mathsf{nv{\text{-}}recall}$, Equation \ref{['eq:recall']}). This metric only counts sufficiently long, contiguous spans of near-verbatim text, for which we can conservatively claim extraction of training data (Section \ref{['sec:prelim:extraction:success']}). We extract nearly all of Harry Potter and the Sorcerer's Stone from jailbroken Claude 3.7 Sonnet (BoN $N=258$, $\mathsf{nv{\text{-}}recall}=95.8\%$). GPT-4.1 requires more jailbreaking attempts ($N=5179$) and refuses to continue after reaching the end of the first chapter; the generated text has $\mathsf{nv{\text{-}}recall}=4.0\%$ with the full book. We extract substantial proportions of the book from Gemini 2.5 Pro and Grok 3 ($76.8\%$ and $70.3\%$, respectively), and notably do not need to jailbreak them to do so ($N=0$). Note: We do not claim we maximized possible extraction for each LLM. Different runs use different underlying generation configurations per LLM.
• Figure 2: Phase 1 of our two-phase procedure. We illustrate Phase 1 (Section \ref{['sec:prelim:bon']}) for Harry Potter and the Sorcerer's Stone: providing an initial instruction to complete a short prefix of ground-truth text from the book. Gemini 2.5 Pro and Grok 3 comply directly (left); for Claude 3.7 Sonnet and GPT-4.1, we use the use Best-of-$N$ jailbreak (right). We evaluate if the production LLM produces a loose approximation of the the suffix using similarity score $s$ (Equation \ref{['eq:phase1-sim']}). If successful ($s\geq0.6$), we proceed to Phase 2 (Figure \ref{['fig:phase2']}, Section \ref{['sec:prelim:extraction']}).
• Figure 3: Phase 2 of our two-phase procedure. If Phase 1 succeeds (i.e., returns a response with $s\geq0.6$, see Figure \ref{['fig:phase1']}, Section \ref{['sec:prelim:bon']}), we proceed to Phase 2 (Section \ref{['sec:prelim:extraction']}). We similarly illustrate Phase 2 for Harry Potter and the Sorcerer's Stone: we repeatedly query to continue the text, until the LLM responds with a refusal or a stop phrase, or we exhaust a specified query budget. Phase 2 culminates in a long-form generation that we compare to a corresponding reference book to gauge extraction success using $\mathsf{nv{\text{-}}recall}$ (Equation \ref{['eq:recall']}, Section \ref{['sec:prelim:extraction:success']}). The prefix in Phase 1 is the only ground-truth text that we provide in the entire two-phase procedure; any additional text that we recover from a book in Phase 2 is generated and returned by the production LLM.
• Figure 4: Near-verbatim block formation. After identifying verbatim blocks, we merge closely aligned, nearby blocks (Equation \ref{['eq:nv-set:taus']}). In both subfigures, the blocks are aligned ($| \Delta^{(B)}_k - \Delta^{(G)}_k | = 0$). The first merge (M1) is very stringent, with a maximum gap $\tau_{\mathrm{gap}}^{(1)}=2$ words, and then filter 1 (F1) only retains blocks that are at least $20$ words long ($l^{(1)}=20$). The second merge (M2), performed on the blocks retained after F1, is slightly more relaxed ($\tau_{\mathrm{gap}}^{(2)}=10$), and so the second filter is more stringent ($l^{(2)}=100$). In Figure \ref{['fig:frankenstein-merge']}, M1 merges very close blocks. The remaining blocks---block 1, block 2* (=block 2 + block 3 + block 4 + block 5), and block 6---are each long enough to be retained by F1 (but note that they would not at this point be retained by F2). These blocks are merged in M2, resulting in a single $141$-word block that is retained after F2. In Figure \ref{['fig:gemini-davinci']}, no blocks are retained. There are verbatim-matching blocks returned by the identify step, but they are too short to be valid evidence for extraction. Our two-pass merge-and-filter procedure removes them; they are not counted in our extraction metric, $m$ (Equation \ref{['eq:matched']}). See Appendix \ref{['app:sec:extraction_details']} for more details.
• Figure 5: Proportion of book extracted ($\mathsf{nv{\text{-}}recall}$). We show $\mathsf{nv{\text{-}}recall}$ (%) for the twelve books for which we run Phase 2. Each bar is annotated with the corresponding $\mathsf{nv{\text{-}}recall}$ for a production LLM-book pair; the number in parentheses above is the BoN samples $N$ in Phase 1 ($N=0$ for Gemini 2.5 Pro and Grok 3, since we do not jailbreak those production LLMs.) $\dagger$ denotes that Phase 1 failed; $*$ indicates we did not attempt Phase 2. Gray shading indicates public domain books. The vertical axis in each row has a different scale. Note: Each bar reflects a single run of Phase 2, where the underlying generation configuration is fixed per LLM but varies across LLMs. The groups of bars do not reflect comparisons of results obtained from testing all production LLMs under the same conditions.