Representation Purification for End-to-End Speech Translation

TL;DR

The paper tackles the domain gap in end-to-end speech translation by identifying content-agnostic speech factors (timbre, pitch, rhythm) that hinder cross-modal transfer and translating these insights into a purification framework. SRPSE jointly learns a content-agnostic encoder and a complex-information encoder, then uses Orthogonal Projection Purification (OPP) to remove noncontent information from the speech representation, augmented by a variational mutual-information objective (vCLUB) and supervision-enhanced perturbations to strengthen robustness. Empirical results on MuST-C and CoVoST-2 show consistent BLEU gains across directions, including strong performance in transcript-free settings, with additional benefits in robustness and cross-modal alignment. The approach reduces reliance on transcription data and presents a practical, scalable path to improved MT-ST knowledge transfer, albeit with some computational overhead and opportunities for richer disentanglement and integration with multimodal LLMs.

Abstract

Speech-to-text translation (ST) is a cross-modal task that involves converting spoken language into text in a different language. Previous research primarily focused on enhancing speech translation by facilitating knowledge transfer from machine translation, exploring various methods to bridge the gap between speech and text modalities. Despite substantial progress made, factors in speech that are not relevant to translation content, such as timbre and rhythm, often limit the efficiency of knowledge transfer. In this paper, we conceptualize speech representation as a combination of content-agnostic and content-relevant factors. We examine the impact of content-agnostic factors on translation performance through preliminary experiments and observe a significant performance deterioration when content-agnostic perturbations are introduced to speech signals. To address this issue, we propose a \textbf{S}peech \textbf{R}epresentation \textbf{P}urification with \textbf{S}upervision \textbf{E}nhancement (SRPSE) framework, which excludes the content-agnostic components within speech representations to mitigate their negative impact on ST. Experiments on MuST-C and CoVoST-2 datasets demonstrate that SRPSE significantly improves translation performance across all translation directions in three settings and achieves preeminent performance under a \textit{transcript-free} setting.

Figures (10)

• Figure 1: BLEU scores on MuST-C En-De dev subsets. VC and raw denote the BLEU scores are calculated with voice-converted audio $\tilde{\mathbf{s}}$ and raw audio $\mathbf{s}$, respectively. The Green bar denotes the G value.
• Figure 2: Averaged $G$ values of each T-Enc layer's outputs.
• Figure 3: Averaged information entropy of cross-attention weights.
• Figure 4: Overview of our proposed framework. The text embedding and MT forward path are deprecated during inference or training in the transcript-free setting.
• Figure 5: Diagram of OPP Module. It consists of two classifiers and an orthogonal projection layer.