TriniMark: A Robust Generative Speech Watermarking Method for Trinity-Level Attribution

TL;DR

TriniMark addresses IP protection and attribution for AI-generated speech by enabling Trinity-level traceability across the content, model, and user. It introduces a two-stage framework: first pre-trains a structure-lightweight watermark encoder/decoder to embed and recover watermark priors in the time-domain, then performs waveform-guided fine-tuning of DDPM-based vocoders to transfer watermark knowledge to generated speech. The diffusion-forward process is described by $q(s_t|s_{t-1})=\mathcal{N}(s_t;\sqrt{1-\beta_t}s_{t-1},\beta_t I)$ with denoising guided by $\epsilon_\theta$, and watermark recovery uses $\mathcal{L}_{simple}$ alongside $\mathcal{L}_{Mel}$ and $\mathcal{L}_{Mag}$. Empirically, TriniMark achieves high fidelity up to 500 bps and demonstrates strong robustness to both individual and compound attacks, outperforming several state-of-the-art baselines in real-world scenarios and enabling reliable attribution across model, content, and end-user levels.

Abstract

The emergence of diffusion models has facilitated the generation of speech with reinforced fidelity and naturalness. While deepfake detection technologies have manifested the ability to identify AI-generated content, their efficacy decreases as generative models become increasingly sophisticated. Furthermore, current research in the field has not adequately addressed the necessity for robust watermarking to safeguard the intellectual property rights associated with synthetic speech and generative models. To remedy this deficiency, we propose a \textbf{ro}bust generative \textbf{s}peech wat\textbf{e}rmarking method (TriniMark) for authenticating the generated content and safeguarding the copyrights by enabling the traceability of the diffusion model. We first design a structure-lightweight watermark encoder that embeds watermarks into the time-domain features of speech and reconstructs the waveform directly. A temporal-aware gated convolutional network is meticulously designed in the watermark decoder for bit-wise watermark recovery. Subsequently, the waveform-guided fine-tuning strategy is proposed for fine-tuning the diffusion model, which leverages the transferability of watermarks and enables the diffusion model to incorporate watermark knowledge effectively. When an attacker trains a surrogate model using the outputs of the target model, the embedded watermark can still be learned by the surrogate model and correctly extracted. Comparative experiments with state-of-the-art methods demonstrate the superior robustness of our method, particularly in countering compound attacks.

Figures (4)

• Figure 1: Different types of watermarking methods and their traceability functions. Post-hoc watermarking methods embed watermarks directly into speech and can only support content-level traceability. Generative watermarking guides the model in synthesizing watermarked speech through watermark features, enabling traceability at both the model and content levels. In contrast, our method first performs pre-embedding of watermarks into the training data, thereby guiding the model to generate watermarked content, which achieves the trinity traceability.
• Figure 2: The pipeline of proposed TriniMark. 1) The stage of pre-training watermark encoder and decoder. The encoder embeds the watermark into the time-domain features of the speech and reconstructs the waveform to obtain the watermarked speech. The decoder disentangles the watermark features and recovers the watermark. To enhance robustness, a noise layer is applied to the watermarked speech before feeding it into the decoder 2) The stage of fine-tuning the diffusion models. The training speech and watermark are first processed by the pretrained encoder to generate watermarked training speech, which is then multiplied with the Gaussian latent variables obtained through the diffusion process to serve as input to the diffusion model. The mel-spectrogram is used as a conditional input to the diffusion model to generate watermarked speech, and the pretrained decoder is then utilized to extract the watermark.
• Figure 3: The Detailed Architecture of Watermark Encoder and Decoder.
• Figure 4: Comparison of Robustness Against Noise-level Attacks. For Gaussian and PN+GN, four different noise levels of Gaussian noise (5, 10, 15, and 20 dB) are set. As the noise level decreases, the attack strength increases. For Pink and GN+PN, four different noise standard deviations (STD) of pink noise (0.1, 0.2, 0.3, and 0.4) are set. As the STD increases, the attack strength increases.