StressTransfer: Stress-Aware Speech-to-Speech Translation with Emphasis Preservation
Xi Chen, Yuchen Song, Satoshi Nakamura
TL;DR
StressTransfer tackles the challenge of preserving lexical emphasis in S2ST by converting source-language emphasis into target-language tags consumed by a controllable TTS. It introduces EmphST-Instruct to generate large-scale, emphasis-aligned S2TT data and EmphST-Bench to rigorously evaluate emphasis preservation, then couples an emphasis-preserving S2TT model with CosyVoice2 to deliver speech that retains focus and intent. The approach demonstrates state-of-the-art performance in emphasis transfer while maintaining strong semantic translation quality, using an LLM-assisted evaluation framework (LLM-as-Judge). The work highlights prosody as a crucial yet under-addressed facet of translation, offering a data-efficient, scalable baseline for expressive S2ST that can be extended across languages and prosodic cues.
Abstract
We propose a stress-aware speech-to-speech translation (S2ST) system that preserves word-level emphasis by leveraging LLMs for cross-lingual emphasis conversion. Our method translates source-language stress into target-language tags that guide a controllable TTS model. To overcome data scarcity, we developed a pipeline to automatically generate aligned training data and introduce the "LLM-as-Judge" for evaluation. Experiments show our approach substantially outperforms baselines in preserving emphasis while maintaining comparable translation quality, speaker intent, and naturalness. Our work highlights the importance of prosody in translation and provides an effective, data-efficient solution for preserving paralinguistic cues in S2ST.