Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Challenge of Identifying the Origin of Black-Box Large Language Models

Ziqing Yang, Yixin Wu, Yun Shen, Wei Dai, Michael Backes, Yang Zhang

TL;DR

The paper addresses the challenge of identifying the origin of black-box LLMs amid widespread unauthorized customization and licensing restrictions. It shows that existing passive AE-based fingerprints and proactive watermarking struggles to robustly identify derivatives, especially after fine-tuning. The authors propose PlugAE, a plug-in watermarking method that optimizes continuous adversarial token embeddings $e^a_{1:k}$ to minimize $L^{e}_{PlugAE}$ and embeds copyright tokens into the tokenizer without altering model weights, supported by theoretical and empirical evidence. This approach provides a practical path to provenance identification and informs policy discussions on IP protection and data governance in large-scale language models.

Abstract

The tremendous commercial potential of large language models (LLMs) has heightened concerns about their unauthorized use. Third parties can customize LLMs through fine-tuning and offer only black-box API access, effectively concealing unauthorized usage and complicating external auditing processes. This practice not only exacerbates unfair competition, but also violates licensing agreements. In response, identifying the origin of black-box LLMs is an intrinsic solution to this issue. In this paper, we first reveal the limitations of state-of-the-art passive and proactive identification methods with experiments on 30 LLMs and two real-world black-box APIs. Then, we propose the proactive technique, PlugAE, which optimizes adversarial token embeddings in a continuous space and proactively plugs them into the LLM for tracing and identification. The experiments show that PlugAE can achieve substantial improvement in identifying fine-tuned derivatives. We further advocate for legal frameworks and regulations to better address the challenges posed by the unauthorized use of LLMs.

The Challenge of Identifying the Origin of Black-Box Large Language Models

TL;DR

The paper addresses the challenge of identifying the origin of black-box LLMs amid widespread unauthorized customization and licensing restrictions. It shows that existing passive AE-based fingerprints and proactive watermarking struggles to robustly identify derivatives, especially after fine-tuning. The authors propose PlugAE, a plug-in watermarking method that optimizes continuous adversarial token embeddings to minimize and embeds copyright tokens into the tokenizer without altering model weights, supported by theoretical and empirical evidence. This approach provides a practical path to provenance identification and informs policy discussions on IP protection and data governance in large-scale language models.

Abstract

The tremendous commercial potential of large language models (LLMs) has heightened concerns about their unauthorized use. Third parties can customize LLMs through fine-tuning and offer only black-box API access, effectively concealing unauthorized usage and complicating external auditing processes. This practice not only exacerbates unfair competition, but also violates licensing agreements. In response, identifying the origin of black-box LLMs is an intrinsic solution to this issue. In this paper, we first reveal the limitations of state-of-the-art passive and proactive identification methods with experiments on 30 LLMs and two real-world black-box APIs. Then, we propose the proactive technique, PlugAE, which optimizes adversarial token embeddings in a continuous space and proactively plugs them into the LLM for tracing and identification. The experiments show that PlugAE can achieve substantial improvement in identifying fine-tuned derivatives. We further advocate for legal frameworks and regulations to better address the challenges posed by the unauthorized use of LLMs.

Paper Structure

This paper contains 17 sections, 3 theorems, 11 equations, 4 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Statement
  4. PlugAE
  5. Limitations of Existing Passive Identification Methods
  6. Method
  7. Experimental Settings
  8. Experimental Results
  9. Ablation Study
  10. Our Vision
  11. Conclusion
  12. Existing Passive Identification Methods
  13. AE-Based Identification Methods
  14. Experimental Settings
  15. Experimental Results
  16. ...and 2 more sections

Key Result

Lemma 4.2

Based on hypothesis:injective, the inverse function $q$ of the embedding layer $g$ is non-differentiable.

Figures (4)

  • Figure 1: A black-box LLM identification scenario.
  • Figure 2: Passive and proactive methods to identify the origin of black-box LLMs.
  • Figure 3: Relationship between TRRs and the model weights' cosine distance with the base model. The inner is a zoomed-in subgraph focusing on the region that excludes counterexamples, offering a more fine-grained view. This reveals a negative correlation between TRRs and cosine distances, suggesting that larger cosine distances are associated with lower TRRs.
  • Figure 4: Influence of the number of tokens on Llama-2-7b.

Theorems & Definitions (7)

  • Lemma 4.2
  • proof
  • Lemma 4.3
  • proof
  • Lemma 4.4
  • proof
  • Claim 4.5