Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Latent Watermarking of Audio Generative Models

Robin San Roman, Pierre Fernandez, Antoine Deleforge, Yossi Adi, Romain Serizel

TL;DR

This work introduces a method that watermarks latent audio generative models by directly watermarking their training data, showing the method to be robust against a broad range of audio edits including filtering, compression or even to changing the model’s decoder, maintaining high detection rates with very few false positives.

Abstract

The advancements in audio generative models have opened up new challenges in their responsible disclosure and the detection of their misuse. In response, we introduce a method to watermark latent generative models by a specific watermarking of their training data. The resulting watermarked models produce latent representations whose decoded outputs are detected with high confidence, regardless of the decoding method used. This approach enables the detection of the generated content without the need for a post-hoc watermarking step. It provides a more secure solution for open-sourced models and facilitates the identification of derivative works that fine-tune or use these models without adhering to their license terms. Our results indicate for instance that generated outputs are detected with an accuracy of more than 75% at a false positive rate of $10^{-3}$, even after fine-tuning the latent generative model.

Latent Watermarking of Audio Generative Models

TL;DR

This work introduces a method that watermarks latent audio generative models by directly watermarking their training data, showing the method to be robust against a broad range of audio edits including filtering, compression or even to changing the model’s decoder, maintaining high detection rates with very few false positives.

Abstract

The advancements in audio generative models have opened up new challenges in their responsible disclosure and the detection of their misuse. In response, we introduce a method to watermark latent generative models by a specific watermarking of their training data. The resulting watermarked models produce latent representations whose decoded outputs are detected with high confidence, regardless of the decoding method used. This approach enables the detection of the generated content without the need for a post-hoc watermarking step. It provides a more secure solution for open-sourced models and facilitates the identification of derivative works that fine-tune or use these models without adhering to their license terms. Our results indicate for instance that generated outputs are detected with an accuracy of more than 75% at a false positive rate of , even after fine-tuning the latent generative model.
Paper Structure (14 sections, 1 figure, 5 tables)

This paper contains 14 sections, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Audio Model Watermarking
  4. Problem statement
  5. Audio watermarking
  6. Audio language model
  7. Experiments
  8. Experimental details
  9. Quality of the audio generative model
  10. Detection and localization results
  11. Attacks on the Model's Watermark
  12. Switching decoder
  13. Model fine-tuning
  14. Conclusion & Discussion

Figures (1)

  • Figure 1: Overview of our method. (1.) We train a watermark generator and detector based on AudioSeal audioseal, enhancing robustness against EnCodec encodec by processing the watermarked audio through EnCodec before detection. (2.) We watermark the audios from our database and train a MusicGen copet2024simple model for next token prediction on this watermarked data. (3.) During inference, we prompt (using text or audio) the language model and decode audios that are detectable with watermark detector.