Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scalable Multilingual Multimodal Machine Translation with Speech-Text Fusion

Yexing Du, Youcheng Pan, Zekun Wang, Zheng Chu, Yichong Huang, Kaiyuan Liu, Bo Yang, Yang Xiang, Ming Liu, Bing Qin

TL;DR

A Speech-guided Machine Translation (SMT) framework that integrates speech and text as fused inputs into an MLLM to improve translation quality and introduces a Self-Evolution Mechanism to mitigate reliance on low-resource data.

Abstract

Multimodal Large Language Models (MLLMs) have achieved notable success in enhancing translation performance by integrating multimodal information. However, existing research primarily focuses on image-guided methods, whose applicability is constrained by the scarcity of multilingual image-text pairs. The speech modality overcomes this limitation due to its natural alignment with text and the abundance of existing speech datasets, which enable scalable language coverage. In this paper, we propose a Speech-guided Machine Translation (SMT) framework that integrates speech and text as fused inputs into an MLLM to improve translation quality. To mitigate reliance on low-resource data, we introduce a Self-Evolution Mechanism. The core components of this framework include a text-to-speech model, responsible for generating synthetic speech, and an MLLM capable of classifying synthetic speech samples and iteratively optimizing itself using positive samples. Experimental results demonstrate that our framework surpasses all existing methods on the Multi30K multimodal machine translation benchmark, achieving new state-of-the-art results. Furthermore, on general machine translation datasets, particularly the FLORES-200, it achieves average state-of-the-art performance in 108 translation directions. Ablation studies on CoVoST-2 confirms that differences between synthetic and authentic speech have negligible impact on translation quality. The code and models are released at https://github.com/yxduir/LLM-SRT.

Scalable Multilingual Multimodal Machine Translation with Speech-Text Fusion

TL;DR

A Speech-guided Machine Translation (SMT) framework that integrates speech and text as fused inputs into an MLLM to improve translation quality and introduces a Self-Evolution Mechanism to mitigate reliance on low-resource data.

Abstract

Multimodal Large Language Models (MLLMs) have achieved notable success in enhancing translation performance by integrating multimodal information. However, existing research primarily focuses on image-guided methods, whose applicability is constrained by the scarcity of multilingual image-text pairs. The speech modality overcomes this limitation due to its natural alignment with text and the abundance of existing speech datasets, which enable scalable language coverage. In this paper, we propose a Speech-guided Machine Translation (SMT) framework that integrates speech and text as fused inputs into an MLLM to improve translation quality. To mitigate reliance on low-resource data, we introduce a Self-Evolution Mechanism. The core components of this framework include a text-to-speech model, responsible for generating synthetic speech, and an MLLM capable of classifying synthetic speech samples and iteratively optimizing itself using positive samples. Experimental results demonstrate that our framework surpasses all existing methods on the Multi30K multimodal machine translation benchmark, achieving new state-of-the-art results. Furthermore, on general machine translation datasets, particularly the FLORES-200, it achieves average state-of-the-art performance in 108 translation directions. Ablation studies on CoVoST-2 confirms that differences between synthetic and authentic speech have negligible impact on translation quality. The code and models are released at https://github.com/yxduir/LLM-SRT.
Paper Structure (51 sections, 1 equation, 5 figures, 12 tables)

This paper contains 51 sections, 1 equation, 5 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Modality-Agnostic Hypothesis
  4. Assumption 1.
  5. Overall Design
  6. MLLM Pre-training
  7. ASR.
  8. S2TT.
  9. SMT.
  10. Self-Evolution Mechanism
  11. Stage I: Experience Acquisition
  12. TTS Inference.
  13. Stage II: Experience Refinement
  14. MT and SMT Inference.
  15. Stage III: Model Updating
  16. ...and 36 more sections

Figures (5)

  • Figure 1: Image-Guided vs. Speech-Guided Machine Translation.
  • Figure 2: Overview of Our SMT Framework. The proposed system architecture comprises two core components: (1) MLLM pretraining and (2) Self-Evolution. This framework takes text input, synthesizes speech of the text via a TTS model, and leverages the MLLM to process both text and speech features for higher-quality translation output. Self-evolution mechanism can autonomously generate training data to iteratively optimize the framework.
  • Figure 3: COMET Results by Resource Level, Categorized as Low, Medium, and High. Our model shows an improvement in translation scores, particularly for low-scoring translation directions.
  • Figure 4: Self-Evolution Rounds of spBLEU / COMET (eng$\rightarrow$xx) on FLORES-200 benchmark.
  • Figure 5: Case Study for Under-Translation. Having undergone speech pre-training, MLLMs align text words with speech. The SMT model, which receives this speech-text fusion input, is prevented from ignoring the input text, thereby mitigating omission errors.