Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Copyright Detective: A Forensic System to Evidence LLMs Flickering Copyright Leakage Risks

Guangwei Zhang, Jianing Zhu, Cheng Qian, Neil Gong, Rada Mihalcea, Zhaozhuo Xu, Jingrui He, Jiaqi Ma, Yun Huang, Chaowei Xiao, Bo Li, Ahmed Abbasi, Dongwon Lee, Heng Ji, Denghui Zhang

TL;DR

The paper addresses the challenge of identifying copyright leakage in LLM outputs, where static classification is insufficient due to legal complexity and model uncertainty. It introduces Copyright Detective, a modular forensic system that unifies Content Recall Detection, Persuasive Jailbreak Detection, Knowledge Memorization Detection, Unlearning Detection, and a Legal Cases Display to gather and validate evidence across black-box and white-box settings. The framework employs inference-time scaling and a suite of metrics (e.g., ROUGE, Levenshtein, Jaccard, Min-K% Prob, PCA Shift, FIM, CKA) and demonstrates its utility through case studies on works like The Hobbit, The Great Gatsby, and Pride and Prejudice. This evidence-driven approach enables scalable, transparent auditing and education for responsible deployment of LLMs while highlighting gaps and future directions in legal-theoretic evaluation and cross-version forensics.

Abstract

We present Copyright Detective, the first interactive forensic system for detecting, analyzing, and visualizing potential copyright risks in LLM outputs. The system treats copyright infringement versus compliance as an evidence discovery process rather than a static classification task due to the complex nature of copyright law. It integrates multiple detection paradigms, including content recall testing, paraphrase-level similarity analysis, persuasive jailbreak probing, and unlearning verification, within a unified and extensible framework. Through interactive prompting, response collection, and iterative workflows, our system enables systematic auditing of verbatim memorization and paraphrase-level leakage, supporting responsible deployment and transparent evaluation of LLM copyright risks even with black-box access.

Copyright Detective: A Forensic System to Evidence LLMs Flickering Copyright Leakage Risks

TL;DR

The paper addresses the challenge of identifying copyright leakage in LLM outputs, where static classification is insufficient due to legal complexity and model uncertainty. It introduces Copyright Detective, a modular forensic system that unifies Content Recall Detection, Persuasive Jailbreak Detection, Knowledge Memorization Detection, Unlearning Detection, and a Legal Cases Display to gather and validate evidence across black-box and white-box settings. The framework employs inference-time scaling and a suite of metrics (e.g., ROUGE, Levenshtein, Jaccard, Min-K% Prob, PCA Shift, FIM, CKA) and demonstrates its utility through case studies on works like The Hobbit, The Great Gatsby, and Pride and Prejudice. This evidence-driven approach enables scalable, transparent auditing and education for responsible deployment of LLMs while highlighting gaps and future directions in legal-theoretic evaluation and cross-version forensics.

Abstract

We present Copyright Detective, the first interactive forensic system for detecting, analyzing, and visualizing potential copyright risks in LLM outputs. The system treats copyright infringement versus compliance as an evidence discovery process rather than a static classification task due to the complex nature of copyright law. It integrates multiple detection paradigms, including content recall testing, paraphrase-level similarity analysis, persuasive jailbreak probing, and unlearning verification, within a unified and extensible framework. Through interactive prompting, response collection, and iterative workflows, our system enables systematic auditing of verbatim memorization and paraphrase-level leakage, supporting responsible deployment and transparent evaluation of LLM copyright risks even with black-box access.
Paper Structure (23 sections, 5 figures)

This paper contains 23 sections, 5 figures.

Table of Contents

  1. Introduction
  2. System Description
  3. Forensic Module Design
  4. Content Recall Detection
  5. Persuasive Jailbreak Detection
  6. Knowledge Memorization Detection
  7. Unlearning Detection
  8. Legal Cases Display
  9. Analyses and Case Studies
  10. Related Work
  11. Conclusion & Future Work
  12. Predefined Datasets
  13. Text Similarity Metrics
  14. Structural and Verbatim Replication Metrics
  15. Lexical Reuse and Modification Metrics
  16. ...and 8 more sections

Figures (5)

  • Figure 1: User interface of Copyright Detective, taking "Content Recall Detection" as an example.
  • Figure 2: Copyright Detective: An integrated system for copyright risk assessment in LLMs.
  • Figure 3: Analysis of forensic modules properties for Copyright Detective. Left: Inference scaling exposes more latent memorization risks in LLMs. Middle: Persuasive jailbreaking shifts the risk distribution, making extraction easier. Right: PCA analysis reveals that unlearning methods leave detectable representational traces.
  • Figure 4: Example use cases of Copyright Detective.
  • Figure 5: Best-of-N experimental results in persuasive jailbreak module using GPT-4o-mini.