Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data?

TL;DR

This work introduces ComSpeech, a composite two-pass S2ST model that can seamlessly integrate any pretrained S2TT and TTS models through a CTC-based vocabulary adaptor, enabling direct S2ST with improved translation quality and decoding speed. It further proposes ComSpeech-ZS, a training regime that eliminates the need for parallel speech data by aligning latent representations between the S2TT and TTS pathways, achieving competitive zero-shot S2ST performance on CVSS. Experimental results show that ComSpeech outperforms state-of-the-art two-pass models in supervised settings and narrows the gap to zero-shot performance, while ComSpeech-ZS approaches supervision-level translation quality without parallel speech data and surpasses cascaded baselines. Overall, the method demonstrates efficient reuse of abundant S2TT and TTS resources to advance high-quality direct S2ST with practical deployment implications.

Abstract

Recently proposed two-pass direct speech-to-speech translation (S2ST) models decompose the task into speech-to-text translation (S2TT) and text-to-speech (TTS) within an end-to-end model, yielding promising results. However, the training of these models still relies on parallel speech data, which is extremely challenging to collect. In contrast, S2TT and TTS have accumulated a large amount of data and pretrained models, which have not been fully utilized in the development of S2ST models. Inspired by this, in this paper, we first introduce a composite S2ST model named ComSpeech, which can seamlessly integrate any pretrained S2TT and TTS models into a direct S2ST model. Furthermore, to eliminate the reliance on parallel speech data, we propose a novel training method ComSpeech-ZS that solely utilizes S2TT and TTS data. It aligns representations in the latent space through contrastive learning, enabling the speech synthesis capability learned from the TTS data to generalize to S2ST in a zero-shot manner. Experimental results on the CVSS dataset show that when the parallel speech data is available, ComSpeech surpasses previous two-pass models like UnitY and Translatotron 2 in both translation quality and decoding speed. When there is no parallel speech data, ComSpeech-ZS lags behind \name by only 0.7 ASR-BLEU and outperforms the cascaded models.

Figures (4)

• Figure 1: Model architecture of our proposed ComSpeech. It includes an S2TT model $\mathcal{F}$, a TTS model $\mathcal{G}$, and a vocabulary adaptor $\mathcal{A}$ to connect $F$ and $G$.
• Figure 2: Illustration of training and inference process in the zero-shot learning scenario. Solid lines represent data flow, while dashed lines represent loss calculation. The gray modules do not participate in the computation.
• Figure 3: ASR-BLEU scores on CVSS Fr$\rightarrow$En test set with different amounts of S2TT data.
• Figure 4: ASR-BLEU scores on CVSS Fr$\rightarrow$En test set with different amounts of TTS data.