Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Models

Xi Chen, Songyang Zhang, Qibing Bai, Kai Chen, Satoshi Nakamura

TL;DR

This work investigates LLaST, an LLM-based end-to-end speech translation framework that fuses a speech encoder, a lightweight adaptor, and a decoder-only LLM. It introduces dual-LoRA fine-tuning, ASR-augmentation, and multilingual data augmentation to achieve scalable, high-performance ST, demonstrated by a state-of-the-art 45.1 BLEU on CoVoST-2 Fr→En and strong results across multiple language pairs. Key findings show that Whisper-based encoders and larger LLMs yield substantial gains, with encoder scaling often delivering greater parameter efficiency than decoder scaling. The approach and open release of data, code, and models aim to establish a robust baseline and guide future research in LLM-driven speech translation.

Abstract

We introduces LLaST, a framework for building high-performance Large Language model based Speech-to-text Translation systems. We address the limitations of end-to-end speech translation(E2E ST) models by exploring model architecture design and optimization techniques tailored for LLMs. Our approach includes LLM-based speech translation architecture design, ASR-augmented training, multilingual data augmentation, and dual-LoRA optimization. Our approach demonstrates superior performance on the CoVoST-2 benchmark and showcases exceptional scaling capabilities powered by LLMs. We believe this effective method will serve as a strong baseline for speech translation and provide insights for future improvements of the LLM-based speech translation framework. We release the data, code and models in https://github.com/openaudiolab/LLaST.

LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Models

TL;DR

This work investigates LLaST, an LLM-based end-to-end speech translation framework that fuses a speech encoder, a lightweight adaptor, and a decoder-only LLM. It introduces dual-LoRA fine-tuning, ASR-augmentation, and multilingual data augmentation to achieve scalable, high-performance ST, demonstrated by a state-of-the-art 45.1 BLEU on CoVoST-2 Fr→En and strong results across multiple language pairs. Key findings show that Whisper-based encoders and larger LLMs yield substantial gains, with encoder scaling often delivering greater parameter efficiency than decoder scaling. The approach and open release of data, code, and models aim to establish a robust baseline and guide future research in LLM-driven speech translation.

Abstract

We introduces LLaST, a framework for building high-performance Large Language model based Speech-to-text Translation systems. We address the limitations of end-to-end speech translation(E2E ST) models by exploring model architecture design and optimization techniques tailored for LLMs. Our approach includes LLM-based speech translation architecture design, ASR-augmented training, multilingual data augmentation, and dual-LoRA optimization. Our approach demonstrates superior performance on the CoVoST-2 benchmark and showcases exceptional scaling capabilities powered by LLMs. We believe this effective method will serve as a strong baseline for speech translation and provide insights for future improvements of the LLM-based speech translation framework. We release the data, code and models in https://github.com/openaudiolab/LLaST.
Paper Structure (36 sections, 7 equations, 4 figures, 6 tables)

This paper contains 36 sections, 7 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Cascaded Speech Translation
  4. End-to-End Speech Translation
  5. LLM-based Speech Translation
  6. Method
  7. Problem Setting
  8. Model Architecture
  9. Speech Encoder
  10. Adaptor
  11. Large Language Model
  12. Training and Inference
  13. Optimization with Dual-LoRA Fintuning
  14. Training with ASR-augmentation
  15. Inference Methodology
  16. ...and 21 more sections

Figures (4)

  • Figure 1: Model Architecture of LLaST We introduce dual-LoRA in the optimization, and keep weights of the speech encoder and LLM frozen. We use a 3-layer MPLs for adaptor and fine-tune its parameters together with dual-LoRA.
  • Figure 2: An example for training data.
  • Figure 3: Influence of different language models. We use Whisper-large-v2 as speech encoder and report SacreBLEU scores on CoVoST-2 test set for all experiments.
  • Figure 4: Influence of different LLMs and ASR-augmentation. We report SacreBLEU scores on CoVoST-2 test set for all experiments.