Speech Translation Refinement using Large Language Models

TL;DR

The paper tackles the challenge of improving speech translation by exploiting large language models to jointly refine ASR transcription and ST output. It introduces the Refine_Both task and three refinement modalities—in-context learning, context-agnostic fine-tuning, and context-aware fine-tuning—and evaluates them on MuST-C and CoVoST 2 using GPT-3.5-turbo, LLaMA3-8B, and Mistral-12B, with CRESS and SpeechLM-P as ST baselines. Key findings show that joint transcription-translation refinement yields larger BLEU and COMET gains and lowers WER, with document-level context (K=3) providing further improvements, and a two-stage, context-aware fine-tuning strategy delivering the strongest performance. The work demonstrates robust improvements across multiple language pairs and datasets and releases code and data on GitHub, highlighting practical potential for enhancing cross-lingual speech translation in real-world scenarios.

Abstract

Recent advancements in large language models (LLMs) have demonstrated their remarkable capabilities across various language tasks. Inspired by the success of text-to-text translation refinement, this paper investigates how LLMs can improve the performance of speech translation by introducing a joint refinement process. Through the joint refinement of speech translation (ST) and automatic speech recognition (ASR) transcription via LLMs, the performance of the ST model is significantly improved in both training-free in-context learning and parameter-efficient fine-tuning scenarios. Additionally, we explore the effect of document-level context on refinement under the context-aware fine-tuning scenario. Experimental results on the MuST-C and CoVoST 2 datasets, which include seven translation tasks, demonstrate the effectiveness of the proposed approach using several popular LLMs including GPT-3.5-turbo, LLaMA3-8B, and Mistral-12B. Further analysis further suggests that jointly refining both transcription and translation yields better performance compared to refining translation alone. Meanwhile, incorporating document-level context significantly enhances refinement performance. We release our code and datasets on GitHub.

Figures (5)

• Figure 1: Illustration of automatic transcription and translation from ASR and ST models. In this example, errors (Bold text) in both the automatic transcription and translation can be potentially corrected mutually.
• Figure 2: Pipeline for the joint refinement. Refine$_{Both}$ is on the right part of the pipeline (highlighted in gray).
• Figure 3: Prompts for ST refinement using LLMs, including instruction, optional in-context examples (used only for in-context learning), and query. The placeholders "< var>" are replaced with their corresponding content. See Appendix \ref{['apdx:prompt_example']} for detailed examples.
• Figure 4: Prompt used in the first stage of fine-tuning.
• Figure 5: Prompt for GPT evaluation.