SHIELD: Evaluation and Defense Strategies for Copyright Compliance in LLM Text Generation

TL;DR

This paper addresses copyright compliance in LLM text generation by proposing SHIELD, a unified framework that couples a curated copyright-aware benchmark with a lightweight, agent-based defense. The defense combines a Copyright Material Detector, a Copyright Status Verifier, and a Copyright Status Guide to detect, verify, and steer generation away from copyrighted content in real time, without requiring model retraining. Empirical results demonstrate that many LLMs generate copyrighted text and that jailbreaking can increase this leakage; SHIELD markedly reduces copyrighted outputs and raises refusal rates, often achieving near-total refusals on API models. The work offers a practical, scalable approach for protecting intellectual property in LLM deployments, including datasets, metrics, and open-source code to enable further adoption and refinement.

Abstract

Large Language Models (LLMs) have transformed machine learning but raised significant legal concerns due to their potential to produce text that infringes on copyrights, resulting in several high-profile lawsuits. The legal landscape is struggling to keep pace with these rapid advancements, with ongoing debates about whether generated text might plagiarize copyrighted materials. Current LLMs may infringe on copyrights or overly restrict non-copyrighted texts, leading to these challenges: (i) the need for a comprehensive evaluation benchmark to assess copyright compliance from multiple aspects; (ii) evaluating robustness against safeguard bypassing attacks; and (iii) developing effective defense targeted against the generation of copyrighted text. To tackle these challenges, we introduce a curated dataset to evaluate methods, test attack strategies, and propose lightweight, real-time defense to prevent the generation of copyrighted text, ensuring the safe and lawful use of LLMs. Our experiments demonstrate that current LLMs frequently output copyrighted text, and that jailbreaking attacks can significantly increase the volume of copyrighted output. Our proposed defense mechanism significantly reduces the volume of copyrighted text generated by LLMs by effectively refusing malicious requests. Code is publicly available at https://github.com/xz-liu/SHIELD

Figures (11)

• Figure 1: An example of LLM outputting copyrighted texts or overprotection.
• Figure 2: An example of different defense mechanisms on LLaMA 3. The first box shows the user prompt. The second box shows the text generated by the original model, the third box shows the text generated by the model with MemFree decoding, and the fourth box shows the refusal response of the model with our Agent-based defense mechanism. The copied text is shown in purple , and the hallucinated text is shown in red. We can depict that while the model with MemFree decoding generates less copied text than the original model, it suffers from hallucination. On the countrary, the model with our Agent-based defense mechanism refuses to generate the copyrighted text, which is the desired behavior.
• Figure 3: The architecture of our SHIELD Defense Mechanism.
• Figure 4: The few-shot examples used by our SHIELD Defense Mechanism.
• Figure 5: Another example of different defense mechanisms on LLaMA 3. The first box shows the user prompt. The second box shows the text generated by the original model, the third box shows the text generated by the model with MemFree decoding, and the fourth box shows the refusal response of the model with our Agent-based defense mechanism. The copied text is shown in purple , and the hallucinated text is shown in red. We can depict that while the model with MemFree decoding generates less copied text than the original model, it suffers from hallucination. On the contrary, the model with our Agent-based defense mechanism refuses to generate the copyrighted text, which is the desired behavior.