Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WMCodec: End-to-End Neural Speech Codec with Deep Watermarking for Authenticity Verification

Junzuo Zhou, Jiangyan Yi, Yong Ren, Jianhua Tao, Tao Wang, Chu Yuan Zhang

TL;DR

WMCodec tackles spoofing by embedding a numerical watermark into speech before compression and extracting it after reconstruction, enabling authenticity verification of neural codecs. It jointly optimizes watermark embedding and speech encoding in an end-to-end framework, employing an iterative Attention Imprint Unit to fuse watermark and speech latent features $z_s$ and $z_w$ and using RVQ for quantization. Experiments on LibriTTS show WMCodec outperforms baselines in watermark imperceptibility and extraction accuracy across bandwidths, achieving over 99% extraction accuracy at 6 kbps with 16 bps capacity under common attacks. This approach enables verification protected neural codecs with higher reliability and practical robustness for real-world deployment.

Abstract

Recent advances in speech spoofing necessitate stronger verification mechanisms in neural speech codecs to ensure authenticity. Current methods embed numerical watermarks before compression and extract them from reconstructed speech for verification, but face limitations such as separate training processes for the watermark and codec, and insufficient cross-modal information integration, leading to reduced watermark imperceptibility, extraction accuracy, and capacity. To address these issues, we propose WMCodec, the first neural speech codec to jointly train compression-reconstruction and watermark embedding-extraction in an end-to-end manner, optimizing both imperceptibility and extractability of the watermark. Furthermore, We design an iterative Attention Imprint Unit (AIU) for deeper feature integration of watermark and speech, reducing the impact of quantization noise on the watermark. Experimental results show WMCodec outperforms AudioSeal with Encodec in most quality metrics for watermark imperceptibility and consistently exceeds both AudioSeal with Encodec and reinforced TraceableSpeech in extraction accuracy of watermark. At bandwidth of 6 kbps with a watermark capacity of 16 bps, WMCodec maintains over 99% extraction accuracy under common attacks, demonstrating strong robustness.

WMCodec: End-to-End Neural Speech Codec with Deep Watermarking for Authenticity Verification

TL;DR

WMCodec tackles spoofing by embedding a numerical watermark into speech before compression and extracting it after reconstruction, enabling authenticity verification of neural codecs. It jointly optimizes watermark embedding and speech encoding in an end-to-end framework, employing an iterative Attention Imprint Unit to fuse watermark and speech latent features and and using RVQ for quantization. Experiments on LibriTTS show WMCodec outperforms baselines in watermark imperceptibility and extraction accuracy across bandwidths, achieving over 99% extraction accuracy at 6 kbps with 16 bps capacity under common attacks. This approach enables verification protected neural codecs with higher reliability and practical robustness for real-world deployment.

Abstract

Recent advances in speech spoofing necessitate stronger verification mechanisms in neural speech codecs to ensure authenticity. Current methods embed numerical watermarks before compression and extract them from reconstructed speech for verification, but face limitations such as separate training processes for the watermark and codec, and insufficient cross-modal information integration, leading to reduced watermark imperceptibility, extraction accuracy, and capacity. To address these issues, we propose WMCodec, the first neural speech codec to jointly train compression-reconstruction and watermark embedding-extraction in an end-to-end manner, optimizing both imperceptibility and extractability of the watermark. Furthermore, We design an iterative Attention Imprint Unit (AIU) for deeper feature integration of watermark and speech, reducing the impact of quantization noise on the watermark. Experimental results show WMCodec outperforms AudioSeal with Encodec in most quality metrics for watermark imperceptibility and consistently exceeds both AudioSeal with Encodec and reinforced TraceableSpeech in extraction accuracy of watermark. At bandwidth of 6 kbps with a watermark capacity of 16 bps, WMCodec maintains over 99% extraction accuracy under common attacks, demonstrating strong robustness.
Paper Structure (15 sections, 7 equations, 2 figures, 2 tables)

This paper contains 15 sections, 7 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. End to End Framework of WMCodec
  4. Attention Imprint Unit
  5. Training Paradigm
  6. Experiments
  7. Experiment Setup
  8. Dataset
  9. Baseline
  10. Training details
  11. Experiment Results and Analysis
  12. Performance of Watermark imperceptibility
  13. Performance of Watermark Extraction Accuracy and Robustness
  14. Ablation Analysis
  15. Conclusion

Figures (2)

  • Figure 1: Example of Watermark as Verification Marking for Codec Protection
  • Figure 2: Overview of the WMCodec framework.