Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Measuring Copyright Risks of Large Language Model via Partial Information Probing

Weijie Zhao, Huajie Shao, Zhaozhuo Xu, Suzhen Duan, Denghui Zhang

TL;DR

This paper introduces Partial Information Probing and Iterative Prompting to assess the copyright infringement risks of Large Language Models (LLMs) when given partial copyrighted text. By constructing datasets from novels, news, and lyrics and measuring output overlap with Rouge-L, the authors demonstrate that larger models and longer outputs increase the likelihood of infringing reproductions, with notable variation across text types and model families. They show that iterative prompting can amplify infringing content, though outputs tend to diverge after several iterations, revealing a complex interaction between model capacity, safety mechanisms, and input type. The work highlights key factors driving infringement risk and provides a framework for evaluating copyright exposure in LLMs, informing policy and risk-management considerations for developers and users.

Abstract

Exploring the data sources used to train Large Language Models (LLMs) is a crucial direction in investigating potential copyright infringement by these models. While this approach can identify the possible use of copyrighted materials in training data, it does not directly measure infringing risks. Recent research has shifted towards testing whether LLMs can directly output copyrighted content. Addressing this direction, we investigate and assess LLMs' capacity to generate infringing content by providing them with partial information from copyrighted materials, and try to use iterative prompting to get LLMs to generate more infringing content. Specifically, we input a portion of a copyrighted text into LLMs, prompt them to complete it, and then analyze the overlap between the generated content and the original copyrighted material. Our findings demonstrate that LLMs can indeed generate content highly overlapping with copyrighted materials based on these partial inputs.

Measuring Copyright Risks of Large Language Model via Partial Information Probing

TL;DR

This paper introduces Partial Information Probing and Iterative Prompting to assess the copyright infringement risks of Large Language Models (LLMs) when given partial copyrighted text. By constructing datasets from novels, news, and lyrics and measuring output overlap with Rouge-L, the authors demonstrate that larger models and longer outputs increase the likelihood of infringing reproductions, with notable variation across text types and model families. They show that iterative prompting can amplify infringing content, though outputs tend to diverge after several iterations, revealing a complex interaction between model capacity, safety mechanisms, and input type. The work highlights key factors driving infringement risk and provides a framework for evaluating copyright exposure in LLMs, informing policy and risk-management considerations for developers and users.

Abstract

Exploring the data sources used to train Large Language Models (LLMs) is a crucial direction in investigating potential copyright infringement by these models. While this approach can identify the possible use of copyrighted materials in training data, it does not directly measure infringing risks. Recent research has shifted towards testing whether LLMs can directly output copyrighted content. Addressing this direction, we investigate and assess LLMs' capacity to generate infringing content by providing them with partial information from copyrighted materials, and try to use iterative prompting to get LLMs to generate more infringing content. Specifically, we input a portion of a copyrighted text into LLMs, prompt them to complete it, and then analyze the overlap between the generated content and the original copyrighted material. Our findings demonstrate that LLMs can indeed generate content highly overlapping with copyrighted materials based on these partial inputs.
Paper Structure (25 sections, 3 equations, 8 figures, 8 tables)

This paper contains 25 sections, 3 equations, 8 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Copyright Law in the United States
  4. LLMs and Copyright
  5. Methods
  6. Partial Information Probing
  7. Evaluation of Influencing Factors
  8. Iterative Prompt
  9. Experiment
  10. LLMs Tested
  11. Datasets
  12. Evaluation Metrics: Rouge Score
  13. Result of Infringing Output
  14. Novel
  15. News
  16. ...and 10 more sections

Figures (8)

  • Figure 1: Using large language models to reproduce novel content could potentially lead to copyright infringement issues.
  • Figure 2: Count of Rouge-L Scores $\geq 0.85$ by LLMs and Parameter Scale. The x-axis represents different LLMs, while the y-axis represents the number of instances where Rouge-L is greater than or equal to 0.85. Each type of LLM is represented by a different color.
  • Figure 3: The average Rouge-L Scores for the three Content Types 'Novel', 'News', and 'Lyrics' can help us compare the performance of different test text types across various models. The x-axis represents different LLMs, while the y-axis represents the average Rouge-L Score for that text type.
  • Figure 4: Rouge-L $\geq 0.85$ for Novels and Lyrics. The x-axis represents different LLMs, while the y-axis represents the number of instances where Rouge-L is greater than or equal to 0.85. The size of different colors in the stacked chart represents the number of instances where Rouge-L $\geq 0.85$.
  • Figure 5: Comparison of Average Rouge-L and Count of High Scores by Max Tokens. The x-axis represents the size of max_tokens. The left y-axis represents the average Rouge-L score, while the right y-axis represents the number of instances where Rouge-L $\geq 0.85$. The average Rouge-L is represented in red, and the number of instances where Rouge-L $\geq 0.85$ is represented in blue.
  • ...and 3 more figures