Language translation, and change of accent for speech-to-speech task using diffusion model
Abhishek Mishra, Ritesh Sur Chowdhury, Vartul Bahuguna, Isha Pandey, Ganesh Ramakrishnan
TL;DR
The paper addresses the problem of simultaneous language translation and accent adaptation in speech-to-speech translation (S2ST). It formulates target speech generation as a conditional diffusion process conditioned on source phonemes and target attributes, leveraging GradTTS to produce Mel spectrograms and a diffusion pipeline inspired by text-to-image diffusion. The approach enables joint optimization of translation and accent transfer within a parameter-efficient framework, with empirical results showing strong baseline TTS performance and qualitative support for cross-lingual, multi-accent synthesis. This diffusion-based, unified S2ST framework holds promise for scalable, expressive cross-lingual communication across languages and accents, while highlighting challenges in cascading errors and computational demands that future work should address.
Abstract
Speech-to-speech translation (S2ST) aims to convert spoken input in one language to spoken output in another, typically focusing on either language translation or accent adaptation. However, effective cross-cultural communication requires handling both aspects simultaneously - translating content while adapting the speaker's accent to match the target language context. In this work, we propose a unified approach for simultaneous speech translation and change of accent, a task that remains underexplored in current literature. Our method reformulates the problem as a conditional generation task, where target speech is generated based on phonemes and guided by target speech features. Leveraging the power of diffusion models, known for high-fidelity generative capabilities, we adapt text-to-image diffusion strategies by conditioning on source speech transcriptions and generating Mel spectrograms representing the target speech with desired linguistic and accentual attributes. This integrated framework enables joint optimization of translation and accent adaptation, offering a more parameter-efficient and effective model compared to traditional pipelines.