CopyBench: Measuring Literal and Non-Literal Reproduction of Copyright-Protected Text in Language Model Generation

Abstract

Evaluating the degree of reproduction of copyright-protected content by language models (LMs) is of significant interest to the AI and legal communities. Although both literal and non-literal similarities are considered by courts when assessing the degree of reproduction, prior research has focused only on literal similarities. To bridge this gap, we introduce CopyBench, a benchmark designed to measure both literal and non-literal copying in LM generations. Using copyrighted fiction books as text sources, we provide automatic evaluation protocols to assess literal and non-literal copying, balanced against the model utility in terms of the ability to recall facts from the copyrighted works and generate fluent completions. We find that, although literal copying is relatively rare, two types of non-literal copying -- event copying and character copying -- occur even in models as small as 7B parameters. Larger models demonstrate significantly more copying, with literal copying rates increasing from 0.2\% to 10.5\% and non-literal copying from 2.3\% to 5.9\% when comparing Llama3-8B and 70B models, respectively. We further evaluate the effectiveness of current strategies for mitigating copying and show that (1) training-time alignment can reduce literal copying but may increase non-literal copying, and (2) current inference-time mitigation methods primarily reduce literal but not non-literal copying.

Figures (14)

• Figure 1: Two categories of reproduction of copyrighted content and two categories of model utility, considered in CopyBench. We also show the text generated by Llama3 70B llama3modelcard given the prompt.
• Figure 2: Scatter plots comparing different models on literal copying, non-literal copying (including event and character copying), and fact recall: (a) smaller models can generate events similar to those found in copyrighted works, (b) a strong correlation exists between copying behaviors and fact recall, (c) mitigation methods reduce literal copying but are less effective for non-literal copying, and (d) a decrease in fact recall is observed in some models and mitigation methods.
• Figure 3: Demonstration of non-literal copying evaluation. We show two LM-generated stories and referenced events and character in the novel Harry Potter and the Sorcerer's Stone (1997). The overlapping events are manually highlighted in red and labeled with their indices. Additionally, the overlapping character names are in bold.
• Figure 4: The skewed distribution of ROUGE-L, Event Overlap (i.e., Event Recall), and Character Overlap (i.e., Character Recall) in literal and non-literal copying evaluation. Specifically, the Llama3-70B llama3modelcard model exhibits a longer tail compared to the other two models, suggesting a higher number of instances with high similarity to copyrighted material.
• Figure 5: Three prompt templates for generating passage completion to evaluate literal copying.