Direct Speech to Speech Translation: A Review

TL;DR

The paper surveys direct speech-to-speech translation (S2ST) as an alternative to cascade ASR-MT-TTS pipelines, highlighting how end-to-end models can reduce latency and preserve speaker voice but face data scarcity and high computational costs. It methodically reviews data sources, feature extraction methods, and a wide spectrum of ML models (from Transformer-based seq2seq to diffusion-based and multimodal architectures) that enable direct S2ST, with emphasis on discrete unit representations (HuBERT-based units), self-supervised learning, and cross-lingual adaptations. A comprehensive taxonomy of datasets, benchmarks, and ethics is provided, along with a detailed enumeration of challenges such as data scarcity, error propagation, speaker variability, and zero-shot generalization. The paper also outlines promising future directions, including expanded language coverage, larger and better-annotated corpora, more efficient architectures, pre-training strategies, and robust evaluation metrics to advance real-time, multilingual communication technologies.

Abstract

Speech to speech translation (S2ST) is a transformative technology that bridges global communication gaps, enabling real time multilingual interactions in diplomacy, tourism, and international trade. Our review examines the evolution of S2ST, comparing traditional cascade models which rely on automatic speech recognition (ASR), machine translation (MT), and text to speech (TTS) components with newer end to end and direct speech translation (DST) models that bypass intermediate text representations. While cascade models offer modularity and optimized components, they suffer from error propagation, increased latency, and loss of prosody. In contrast, direct S2ST models retain speaker identity, reduce latency, and improve translation naturalness by preserving vocal characteristics and prosody. However, they remain limited by data sparsity, high computational costs, and generalization challenges for low-resource languages. The current work critically evaluates these approaches, their tradeoffs, and future directions for improving real time multilingual communication.

Figures (4)

• Figure 1: The adopted PRISMA methodology
• Figure 2: Various countries along with their counts of works
• Figure 3: Classifications used in the current study
• Figure 4: Visualization of number of papers with its datasets