Leveraging Unit Language Guidance to Advance Speech Modeling in Textless Speech-to-Speech Translation

TL;DR

This work tackles textless speech-to-speech translation by introducing a unit language, a text-like transcription produced via unsupervised n-gram language modeling, to guide cross-modal and cross-lingual learning. It integrates unit-language guided multi-task training with dedicated source/target unit-language decoders and introduces task prompt modeling to resolve CM/CL conflicts, achieving state-of-the-art performance on VoxPopuli for textless S2ST. Empirical results show that unit language can closely match text-based performance, with CL providing larger gains than CM and prompts enabling harmonious joint training across tasks. The findings suggest unit language as a viable surrogate for text in speech modeling, particularly beneficial for languages with limited or no written forms.

Abstract

The success of building textless speech-to-speech translation (S2ST) models has attracted much attention. However, S2ST still faces two main challenges: 1) extracting linguistic features for various speech signals, called cross-modal (CM), and 2) learning alignment of difference languages in long sequences, called cross-lingual (CL). We propose the unit language to overcome the two modeling challenges. The unit language can be considered a text-like representation format, constructed using $n$-gram language modeling. We implement multi-task learning to utilize the unit language in guiding the speech modeling process. Our initial results reveal a conflict when applying source and target unit languages simultaneously. We propose task prompt modeling to mitigate this conflict. We conduct experiments on four languages of the Voxpupil dataset. Our method demonstrates significant improvements over a strong baseline and achieves performance comparable to models trained with text.

Figures (11)

• Figure 1: The proposed unit language and three other text types represent the transcription. The unit language is created using unsupervised language modeling.
• Figure 2: Architecture of textless S2ST. The dashed line part will be removed during inference. The filterbank features serves as the only input to the network. Blue modules are used to guide the modeling process.
• Figure 3: The guiding process of the task prompt.
• Figure 4: Average lengths of different types of tokens on En-Es and En-Fr training datasets. "Unit_l" denotes the unit language. Note that the units used here have had continuous repetitions removed.
• Figure 5: The influences of CM and CL on Fr-En (left) and En-Fr (right) tasks.