Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bileve: Securing Text Provenance in Large Language Models Against Spoofing with Bi-level Signature

Tong Zhou, Xuandong Zhao, Xiaolin Xu, Shaolei Ren

TL;DR

A bi-level signature scheme, Bileve, which embeds fine-grained signature bits for integrity checks as well as a coarse-grained signal to trace text sources when the signature is invalid (enhancing detectability) via a novel rank-based sampling strategy.

Abstract

Text watermarks for large language models (LLMs) have been commonly used to identify the origins of machine-generated content, which is promising for assessing liability when combating deepfake or harmful content. While existing watermarking techniques typically prioritize robustness against removal attacks, unfortunately, they are vulnerable to spoofing attacks: malicious actors can subtly alter the meanings of LLM-generated responses or even forge harmful content, potentially misattributing blame to the LLM developer. To overcome this, we introduce a bi-level signature scheme, Bileve, which embeds fine-grained signature bits for integrity checks (mitigating spoofing attacks) as well as a coarse-grained signal to trace text sources when the signature is invalid (enhancing detectability) via a novel rank-based sampling strategy. Compared to conventional watermark detectors that only output binary results, Bileve can differentiate 5 scenarios during detection, reliably tracing text provenance and regulating LLMs. The experiments conducted on OPT-1.3B and LLaMA-7B demonstrate the effectiveness of Bileve in defeating spoofing attacks with enhanced detectability. Code is available at https://github.com/Tongzhou0101/Bileve-official.

Bileve: Securing Text Provenance in Large Language Models Against Spoofing with Bi-level Signature

TL;DR

A bi-level signature scheme, Bileve, which embeds fine-grained signature bits for integrity checks as well as a coarse-grained signal to trace text sources when the signature is invalid (enhancing detectability) via a novel rank-based sampling strategy.

Abstract

Text watermarks for large language models (LLMs) have been commonly used to identify the origins of machine-generated content, which is promising for assessing liability when combating deepfake or harmful content. While existing watermarking techniques typically prioritize robustness against removal attacks, unfortunately, they are vulnerable to spoofing attacks: malicious actors can subtly alter the meanings of LLM-generated responses or even forge harmful content, potentially misattributing blame to the LLM developer. To overcome this, we introduce a bi-level signature scheme, Bileve, which embeds fine-grained signature bits for integrity checks (mitigating spoofing attacks) as well as a coarse-grained signal to trace text sources when the signature is invalid (enhancing detectability) via a novel rank-based sampling strategy. Compared to conventional watermark detectors that only output binary results, Bileve can differentiate 5 scenarios during detection, reliably tracing text provenance and regulating LLMs. The experiments conducted on OPT-1.3B and LLaMA-7B demonstrate the effectiveness of Bileve in defeating spoofing attacks with enhanced detectability. Code is available at https://github.com/Tongzhou0101/Bileve-official.
Paper Structure (25 sections, 5 equations, 9 figures, 6 tables, 3 algorithms)

This paper contains 25 sections, 5 equations, 9 figures, 6 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background and Related Works
  3. Language Model Basics
  4. LLM Watermarks
  5. Spoofing Attacks
  6. Potential Attack: Semantic Manipulation
  7. Threat Model
  8. Attack Method
  9. Proposed Defense
  10. Single-level Signature
  11. Bi-level Signature (Bileve)
  12. Experiments
  13. Experimental Setup
  14. Detectability
  15. Generation Quality
  16. ...and 10 more sections

Figures (9)

  • Figure 1: Overview of Bileve.(a) Embedding: The first $m$ tokens from $\mathcal{M}$ form the message, which is signed using a secret key. Candidate tokens are selected via a rank-based strategy employing a Weighted Rank Addition (WRA) score, with a coarse-grained signal embedded. It then embeds the fine-grained signature by choosing the first candidate matching the designated signature bit. (b) Detection: We first extract the message-signature pair to conduct an integrity check using the public key. A statistical test is performed if necessary.
  • Figure 2: The alignment cost of human vs LLM.
  • Figure 3: The detectability of different schemes with OPT-1.3B.
  • Figure 3: The perplexity of applying different schemes to OPT-1.3B
  • Figure 4: The p-value and alignment cost of each segment.
  • ...and 4 more figures