Cross-Modal Robustness Transfer (CMRT): Training Robust Speech Translation Models Using Adversarial Text

TL;DR

This work addresses the vulnerability of End-to-End Speech Translation (E2E-ST) to inflectional morphology by introducing Cross-Modal Robustness Transfer (CMRT), a framework that transfers text-based robustness to the speech modality without generating adversarial speech data. CMRT comprises a two-stage process: CMRT-TR aligns speech and text representations via Word-Aligned Contrastive Learning (WACO) and mixup to produce a strongly shared latent space, followed by CMRT-FN which fine-tunes robustness by injecting adversarial text embeddings into the speech manifold with adversarial mixup and asymmetric KL regularization. A key extension, Speech-MORPHEUS, adapts MORPHEUS inflection perturbations to the speech domain to create challenging adversarial test data, enabling robust evaluation. Across four language directions on CoVoST 2, CMRT-FN yields an average robustness gain exceeding 3 BLEU points over baselines, while avoiding the substantial costs of adversarial speech generation and maintaining better performance on clean data than full speech-based adversarial fine-tuning. This approach establishes a practical baseline for robust E2E-ST and demonstrates that cross-modal robustness can be effectively achieved through shared latent representations and text-only adversarial training.

Abstract

End-to-End Speech Translation (E2E-ST) has seen significant advancements, yet current models are primarily benchmarked on curated, "clean" datasets. This overlooks critical real-world challenges, such as morphological robustness to inflectional variations common in non-native or dialectal speech. In this work, we adapt a text-based adversarial attack targeting inflectional morphology to the speech domain and demonstrate that state-of-the-art E2E-ST models are highly vulnerable it. While adversarial training effectively mitigates such risks in text-based tasks, generating high-quality adversarial speech data remains computationally expensive and technically challenging. To address this, we propose Cross-Modal Robustness Transfer (CMRT), a framework that transfers adversarial robustness from the text modality to the speech modality. Our method eliminates the requirement for adversarial speech data during training. Extensive experiments across four language pairs demonstrate that CMRT improves adversarial robustness by an average of more than 3 BLEU points, establishing a new baseline for robust E2E-ST without the overhead of generating adversarial speech.

Figures (5)

• Figure 1: CMRT aligns speech and text semantic spaces (right). To simulate adversarial speech inflections, clean speech embeddings (e.g., "happiness") are replaced with adversarial text embeddings (e.g., "happy") during robustness fine-tuning.
• Figure 2: An overview illustration of our proposed method. Since our method is composed of two steps, CMRT-TR (§\ref{['sec:bridging_gap']} and §\ref{['sec:cmrt_training']}) followed by CMRT-FN (§\ref{['sec:cmrt_fn']}), we refer to them as TR and FN respectively in the figure.
• Figure 3: The figure shows the correlation between the cosine similarity of speech and text representations on En-De dev set, and the BLEU score on the adversarial test set. We see a correlation between speech and text alignment and CMRT-FN's effectiveness.
• Figure 4: Effect of KL divergence weight $\lambda_{kl}$ on adversarial Morpheus and original CoVoST 2 test sets.
• Figure 5: We compare the CKA similarity of representations of adversarial sentences of CMRT-TR and CMRT-FN models against TTS-Morpheus-FN. CMRT-FN model is more aligned with TTS-Morpheus-FN because both are trained to handle Morpheus errors.