Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PROST-LLM: Progressively Enhancing the Speech-to-Speech Translation Capability in LLMs

Jing Xu, Jiaqi Wang, Daxin Tan, Xiao Chen

TL;DR

PROST-LLM tackles the data scarcity challenge in LLM-based speech-to-speech translation by progressively enhancing S2ST through a three-stage pipeline: supervised fine-tuning on the CVSS corpus with tri-task learning or a chain-of-modality strategy, automatic construction of preference data via self-sampling and back-translation, and a preference optimization step using Direct Preference Optimization. The method eliminates reliance on large-scale human-annotated S2ST data by leveraging monolingual speech resources to generate and evaluate candidate translations, guided by a margin-based objective. Key contributions include the first integration of tri-task/chain learning for S2ST in LLMs, the application of back-translation-driven PO to S2ST, and demonstrations that monolingual data can further boost performance while reducing data requirements. Empirical results show substantial BLEU improvements and a dramatic narrowing of the gap to cascaded systems, with robustness across back-translation metrics and PO algorithms, suggesting practical impact for multilingual S2ST with limited paired data.

Abstract

Although Large Language Models (LLMs) excel in many tasks, their application to Speech-to-Speech Translation (S2ST) is underexplored and hindered by data scarcity. To bridge this gap, we propose PROST-LLM (PROgressive Speech-to-speech Translation) to enhance the S2ST capabilities in LLMs progressively. First, we fine-tune the LLMs with the CVSS corpus, employing designed tri-task learning and chain of modality methods to boost the initial performance. Then, leveraging the fine-tuned model, we generate preference pairs through self-sampling and back-translation without human evaluation. Finally, these preference pairs are used for preference optimization to enhance the model's S2ST capability further. Extensive experiments confirm the effectiveness of our proposed PROST-LLM in improving the S2ST capability of LLMs.

PROST-LLM: Progressively Enhancing the Speech-to-Speech Translation Capability in LLMs

TL;DR

PROST-LLM tackles the data scarcity challenge in LLM-based speech-to-speech translation by progressively enhancing S2ST through a three-stage pipeline: supervised fine-tuning on the CVSS corpus with tri-task learning or a chain-of-modality strategy, automatic construction of preference data via self-sampling and back-translation, and a preference optimization step using Direct Preference Optimization. The method eliminates reliance on large-scale human-annotated S2ST data by leveraging monolingual speech resources to generate and evaluate candidate translations, guided by a margin-based objective. Key contributions include the first integration of tri-task/chain learning for S2ST in LLMs, the application of back-translation-driven PO to S2ST, and demonstrations that monolingual data can further boost performance while reducing data requirements. Empirical results show substantial BLEU improvements and a dramatic narrowing of the gap to cascaded systems, with robustness across back-translation metrics and PO algorithms, suggesting practical impact for multilingual S2ST with limited paired data.

Abstract

Although Large Language Models (LLMs) excel in many tasks, their application to Speech-to-Speech Translation (S2ST) is underexplored and hindered by data scarcity. To bridge this gap, we propose PROST-LLM (PROgressive Speech-to-speech Translation) to enhance the S2ST capabilities in LLMs progressively. First, we fine-tune the LLMs with the CVSS corpus, employing designed tri-task learning and chain of modality methods to boost the initial performance. Then, leveraging the fine-tuned model, we generate preference pairs through self-sampling and back-translation without human evaluation. Finally, these preference pairs are used for preference optimization to enhance the model's S2ST capability further. Extensive experiments confirm the effectiveness of our proposed PROST-LLM in improving the S2ST capability of LLMs.
Paper Structure (17 sections, 1 equation, 1 figure, 5 tables)

This paper contains 17 sections, 1 equation, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Overview
  4. Supervised fine-tuning
  5. Preference data construction
  6. Preference optimization
  7. Experimental setup
  8. Training configuration
  9. Datasets
  10. Evaluation metrics
  11. Experimental results
  12. Comparison systems
  13. Main results
  14. Monolingual vs. S2ST preference data
  15. Influence of back-translation metrics
  16. ...and 2 more sections

Figures (1)

  • Figure 1: (a) Our PROST-LLM training system: (i) Step 1: Supervised fine-tuning (SFT) the LLM. (ii) Step 2: Based on the SFT LLM, we construct preference data pairs (e.g.,$(S_A, S_B^1, S_B^2)$) by comparing back-translated answer pairs $\hat{S}_A^1, \hat{S}_A^2$ with the ground truth $S_A$. (iii) Step 3: Preference optimizing the SFT LLM using the constructed preference data pairs. (b) The architecture of PROST-LLM in step 1.