Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Distortion-Free Watermark for Autoregressive Audio Generation Models

Yihan Wu, Georgios Milis, Ruibo Chen, Heng Huang

TL;DR

Aligned-IS is introduced, a novel, distortion-free watermark, specifically crafted for audio generation models, that utilizes a clustering approach that treats tokens within the same cluster equivalently, effectively countering the retokenization mismatch issue.

Abstract

The rapid advancement of next-token-prediction models has led to widespread adoption across modalities, enabling the creation of realistic synthetic media. In the audio domain, while autoregressive speech models have propelled conversational interactions forward, the potential for misuse, such as impersonation in phishing schemes or crafting misleading speech recordings, has also increased. Security measures such as watermarking have thus become essential to ensuring the authenticity of digital media. Traditional statistical watermarking methods used for autoregressive language models face challenges when applied to autoregressive audio models, due to the inevitable ``retokenization mismatch'' - the discrepancy between original and retokenized discrete audio token sequences. To address this, we introduce Aligned-IS, a novel, distortion-free watermark, specifically crafted for audio generation models. This technique utilizes a clustering approach that treats tokens within the same cluster equivalently, effectively countering the retokenization mismatch issue. Our comprehensive testing on prevalent audio generation platforms demonstrates that Aligned-IS not only preserves the quality of generated audio but also significantly improves the watermark detectability compared to the state-of-the-art distortion-free watermarking adaptations, establishing a new benchmark in secure audio technology applications.

Robust Distortion-Free Watermark for Autoregressive Audio Generation Models

TL;DR

Aligned-IS is introduced, a novel, distortion-free watermark, specifically crafted for audio generation models, that utilizes a clustering approach that treats tokens within the same cluster equivalently, effectively countering the retokenization mismatch issue.

Abstract

The rapid advancement of next-token-prediction models has led to widespread adoption across modalities, enabling the creation of realistic synthetic media. In the audio domain, while autoregressive speech models have propelled conversational interactions forward, the potential for misuse, such as impersonation in phishing schemes or crafting misleading speech recordings, has also increased. Security measures such as watermarking have thus become essential to ensuring the authenticity of digital media. Traditional statistical watermarking methods used for autoregressive language models face challenges when applied to autoregressive audio models, due to the inevitable ``retokenization mismatch'' - the discrepancy between original and retokenized discrete audio token sequences. To address this, we introduce Aligned-IS, a novel, distortion-free watermark, specifically crafted for audio generation models. This technique utilizes a clustering approach that treats tokens within the same cluster equivalently, effectively countering the retokenization mismatch issue. Our comprehensive testing on prevalent audio generation platforms demonstrates that Aligned-IS not only preserves the quality of generated audio but also significantly improves the watermark detectability compared to the state-of-the-art distortion-free watermarking adaptations, establishing a new benchmark in secure audio technology applications.
Paper Structure (41 sections, 1 theorem, 2 equations, 2 figures, 34 tables, 3 algorithms)

This paper contains 41 sections, 1 theorem, 2 equations, 2 figures, 34 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Audio generation models.
  4. Post hoc audio watermarking.
  5. Distortion-free watermark.
  6. Preliminary
  7. Notations.
  8. Statistical Watermarks
  9. Autoregressive audio generation models
  10. Retokenization mismatch.
  11. Methodology
  12. Audio token clustering
  13. Aligned inverse sampling
  14. Inverse Sampling.
  15. Aligned Inverse Sampling.
  16. ...and 26 more sections

Key Result

Theorem 4.1

Aligned inverse sampling is a distortion-free watermark.

Figures (2)

  • Figure 1: Left: Illustration of the retokenization mismatch problem during audio watermark detection. Right: Performance comparison of audio watermark detection on raw versus retokenized audio sequences. We evaluate KGW watermark kirchenbauer2023watermark DiPmark wu2023dipmark, and $\gamma$-reweight hu2023unbiased, reporting the true positive rate at a fixed 1% false positive rate.
  • Figure 2: Audio-quality impact of watermarking methods. Top: FAD scores on the Dolly CW dataset evaluated with SpiritLM. Bottom: FAD scores on the LibriSpeech dataset evaluated with SpiritLM.

Theorems & Definitions (4)

  • Definition 3.1: Distortion-free watermark
  • Theorem 4.1
  • proof
  • Definition 4.2