P2Mark: Plug-and-play Parameter-level Watermarking for Neural Speech Generation

TL;DR

P2Mark presents a plug-and-play parameter-level watermarking approach for neural speech generation by embedding watermark information into released weights via a WM-LoRA adapter. A gradient orthogonal projection optimization (WGOPO) preserves audio quality while maintaining watermark integrity during joint training, enabling flexible watermark configuration without retraining. The method is instantiated in two NSG frameworks (P2Mark-Vocoder and P2Mark-Codec) and demonstrates comparable watermark extraction accuracy, perceptual quality, and robustness to state-of-the-art open-source watermarking techniques, while enabling open-source white-box protection. This work provides a practical path toward proactive tracing and copyright protection for open-source NSG systems, with significant implications for secure deployment and intellectual property protection in AI-generated content.

Abstract

Neural speech generation (NSG) has rapidly advanced as a key component of artificial intelligence-generated content, enabling the generation of high-quality, highly realistic speech for diverse applications. This development increases the risk of technique misuse and threatens social security. Audio watermarking can embed imperceptible marks into generated audio, providing a promising approach for secure NSG usage. However, current audio watermarking methods are mainly applied at the audio-level or feature-level, which are not suitable for open-sourced scenarios where source codes and model weights are released. To address this limitation, we propose a Plug-and-play Parameter-level WaterMarking (P2Mark) method for NSG. Specifically, we embed watermarks into the released model weights, offering a reliable solution for proactively tracing and protecting model copyrights in open-source scenarios. During training, we introduce a lightweight watermark adapter into the pre-trained model, allowing watermark information to be merged into the model via this adapter. This design ensures both the flexibility to modify the watermark before model release and the security of embedding the watermark within model parameters after model release. Meanwhile, we propose a gradient orthogonal projection optimization strategy to ensure the quality of the generated audio and the accuracy of watermark preservation. Experimental results on two mainstream waveform decoders in NSG (i.e., vocoder and codec) demonstrate that P2Mark achieves comparable performance to state-of-the-art audio watermarking methods that are not applicable to open-source white-box protection scenarios, in terms of watermark extraction accuracy, watermark imperceptibility, and robustness.

Figures (5)

• Figure 1: Audio watermarking methods. Existing audio watermarks fall into (a) audio-level watermarking and (b) feature-level watermarking. In the former, watermark embeddings are added to the generated audio; in the latter, watermark embeddings are added to the acoustic features. However, these methods are not applicable in open-source scenarios because users can easily modify the code and remove the watermarks during audio generation. (c) This paper introduces P2Mark, a parameter-level watermarking technique that integrates the watermark into the released model weights, making it impossible for users to delete the watermarks in open-source scenarios.
• Figure 2: The overall framework of the proposed method. First, pre-train the waveform decoder (part a), then train the watermark encoder, watermark decoder, and watermark adapter (part b). Subsequently, different watermarks can be merged with the weights of the adapter and waveform decoder to obtain different instances of models (part c). Finally, these instances of models are released as part of the speech generation models. The speech generated by the open-source model can be detected by the watermark decoder (part d).
• Figure 3: The LoRA Adapter (a) and the Plug-and-play Watermarking Adapter, WM-LoRA Adapter (b).
• Figure 4: The framework of P2Mark-Vocoder and P2Mark-Codec. It consists of three modules: a watermark encoder, a watermark adapter and a watermark decoder. To evaluate the generalizability of our method, we apply P2Mark to two mainstream waveform decoders, vocoder and codec decoder, resulting in P2Mark-Vocoder and P2Mark-Codec. The primary difference between the two lies in the type of acoustic features they use: P2Mark-Vocoder operates on mel-spectrograms, whereas P2Mark-Codec uses acoustic tokens. Therefore, P2Mark-Vocoder includes an additional mel-spectrogram extraction module (a), while P2Mark-Codec incorporates a codec encoder and a quantizer (b).
• Figure 5: Training Optimization Process. The Discriminator Loss $\mathcal{L}_{D}$, Watermark Loss $\mathcal{L}_{wm}$, and Generative Loss $\mathcal{L}_{G}$ are optimized in an alternating manner. Both $\mathcal{L}_{wm}$ and $\mathcal{L}_{G}$ jointly update the Wm-LoRA parameters in the generator, which may lead to conflicting optimization directions. To address this, we propose WGOPO. When the angle between the gradient $g_{\text{gen}}$ (from the generative loss) and the gradient $g_{\text{wm}}$ (from the watermark loss) exceeds $\pi/2$, $g_{\text{gen}}$ is projected onto the plane orthogonal to the saved $g_{\text{wm}}$, ensuring that the watermark loss does not increase during optimization.