Direct Punjabi to English speech translation using discrete units

TL;DR

The paper addresses the scarcity and inefficiency of direct speech-to-speech translation for low-resource languages by introducing Unit-to-Unit Translation (U2UT), a Transformer-based model that translates discrete acoustic units derived from HuBERT for both source and target speech, followed by HiFi-GAN vocoding to synthesize the output. By constructing Punjabi–English parallel speech data through Kathbath and SeamlessM4T, the authors show that discrete-unit input enables a 3.69 BLEU improvement over a spectrogram-based S2UT baseline, though performance remains below cascaded methods due to data limitations. The work contributes a complete end-to-end pipeline, including data preparation, unit-based translation, and unit-to-speech synthesis, and demonstrates the viability of discrete representations for direct S2ST on a low-resource language. This approach promises more efficient, low-latency translation for underserved languages and motivates further data collection and transfer-learning strategies to close the gap with cascaded systems.

Abstract

Speech-to-speech translation is yet to reach the same level of coverage as text-to-text translation systems. The current speech technology is highly limited in its coverage of over 7000 languages spoken worldwide, leaving more than half of the population deprived of such technology and shared experiences. With voice-assisted technology (such as social robots and speech-to-text apps) and auditory content (such as podcasts and lectures) on the rise, ensuring that the technology is available for all is more important than ever. Speech translation can play a vital role in mitigating technological disparity and creating a more inclusive society. With a motive to contribute towards speech translation research for low-resource languages, our work presents a direct speech-to-speech translation model for one of the Indic languages called Punjabi to English. Additionally, we explore the performance of using a discrete representation of speech called discrete acoustic units as input to the Transformer-based translation model. The model, abbreviated as Unit-to-Unit Translation (U2UT), takes a sequence of discrete units of the source language (the language being translated from) and outputs a sequence of discrete units of the target language (the language being translated to). Our results show that the U2UT model performs better than the Speech-to-Unit Translation (S2UT) model by a 3.69 BLEU score.

Figures (3)

• Figure 1: The overall framework of direct speech-to-speech translation using acoustic discrete units: It consists of (1) a Pre-processing step that converts speech to units, (2) a Transformer based Unit to Unit Translation (U2UT) model, (3) a Post-processing step that converts predicted target units to target speech. The dotted lines represent part of the framework that runs only during model training.
• Figure 2: Sample count vs number of discrete units per sample in English and Punjabi subsets of Kathbath dataset
• Figure 3: Validation loss for Exp1. 1 in Table \ref{['tab4']}.