DiffNorm: Self-Supervised Normalization for Non-autoregressive Speech-to-speech Translation

TL;DR

DiffNorm is introduced, a diffusion-based normalization strategy that simplifies data distributions for training NAT models, and is proposed to regularize NATs with classifier-free guidance, improving model robustness and translation quality by randomly dropping out source information during training.

Abstract

Non-autoregressive Transformers (NATs) are recently applied in direct speech-to-speech translation systems, which convert speech across different languages without intermediate text data. Although NATs generate high-quality outputs and offer faster inference than autoregressive models, they tend to produce incoherent and repetitive results due to complex data distribution (e.g., acoustic and linguistic variations in speech). In this work, we introduce DiffNorm, a diffusion-based normalization strategy that simplifies data distributions for training NAT models. After training with a self-supervised noise estimation objective, DiffNorm constructs normalized target data by denoising synthetically corrupted speech features. Additionally, we propose to regularize NATs with classifier-free guidance, improving model robustness and translation quality by randomly dropping out source information during training. Our strategies result in a notable improvement of about +7 ASR-BLEU for English-Spanish (En-Es) and +2 ASR-BLEU for English-French (En-Fr) translations on the CVSS benchmark, while attaining over 14x speedup for En-Es and 5x speedup for En-Fr translations compared to autoregressive baselines.

Figures (6)

• Figure 1: Overview of our proposed system. We first normalize the target speech units with the denoising process from the latent diffusion model. Then speech-to-unit (S2UT) model is trained to predict normalized units, which are converted into waveform from an off-the-shelf unit-vocoder.
• Figure 2: Visualization of our latent diffusion model's denoising process for speech normalization. The clean latent $\bm z_0$ is synthetically noised (into $\bm z_T$) and the reverse diffusion process gradually denoise it to generate normalized speech units.
• Figure 3: Visualization of CMLM for speech-to-unit translation where the model is trained with the unmasking objective to recover $\bm y_{\rm norm}$. When classifier-free guidance is used, with probability $p_{\text{drop}}$, we replace the encoded source speech $\bm g$ by a "null" representation $\bm g_{\emptyset}$.
• Figure 4: Trade-off between quality (ASR-BLEU) and latency for varying numbers of decoding iterations. Five markers correspond to {15, 10, 7, 5, 3} decoding iterations. Decreasing the number of iterations results in a decline in model performance, traded off for faster speedup. With DiffNorm and CG, our S2UT model achieves a better quality-latency trade-off than CMLM and outperforms a strong autoregressive baseline with large speedups.
• Figure 5: Visualization of reconstructed speech's log-mel spectrograms. Noticeable divergence from the original speech is highlighted in the white bounding boxes.