Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Revisiting Direct Speech-to-Text Translation with Speech LLMs: Better Scaling than CoT Prompting?

Oriol Pareras, Gerard I. Gállego, Federico Costa, Cristina España-Bonet, Javier Hernando

TL;DR

The paper investigates how prompting strategy affects end-to-end S2TT as pseudo-labeled data scales. It constructs a large pseudo-labeled S2TT dataset by translating ASR transcriptions into six European languages and trains S2TT models with Direct and CoT prompting. Results show Direct prompting scales more stably with data, while CoT's gains diminish as data increases, with CoT sometimes degrading ASR performance when heavily data-augmented. The findings suggest Direct prompting as a scalable, computation-efficient alternative for S2TT, with potential benefits from richer speech annotations in future work.

Abstract

Recent work on Speech-to-Text Translation (S2TT) has focused on LLM-based models, introducing the increasingly adopted Chain-of-Thought (CoT) prompting, where the model is guided to first transcribe the speech and then translate it. CoT typically outperforms direct prompting primarily because it can exploit abundant Automatic Speech Recognition (ASR) and Text-to-Text Translation (T2TT) datasets to explicitly model its steps. In this paper, we systematically compare CoT and Direct prompting under increasing amounts of S2TT data. To this end, we pseudo-label an ASR corpus by translating its transcriptions into six European languages, and train LLM-based S2TT systems with both prompting strategies at different data scales. Our results show that Direct improves more consistently as the amount of data increases, suggesting that it may become a more effective approach as larger S2TT resources are created.

Revisiting Direct Speech-to-Text Translation with Speech LLMs: Better Scaling than CoT Prompting?

TL;DR

The paper investigates how prompting strategy affects end-to-end S2TT as pseudo-labeled data scales. It constructs a large pseudo-labeled S2TT dataset by translating ASR transcriptions into six European languages and trains S2TT models with Direct and CoT prompting. Results show Direct prompting scales more stably with data, while CoT's gains diminish as data increases, with CoT sometimes degrading ASR performance when heavily data-augmented. The findings suggest Direct prompting as a scalable, computation-efficient alternative for S2TT, with potential benefits from richer speech annotations in future work.

Abstract

Recent work on Speech-to-Text Translation (S2TT) has focused on LLM-based models, introducing the increasingly adopted Chain-of-Thought (CoT) prompting, where the model is guided to first transcribe the speech and then translate it. CoT typically outperforms direct prompting primarily because it can exploit abundant Automatic Speech Recognition (ASR) and Text-to-Text Translation (T2TT) datasets to explicitly model its steps. In this paper, we systematically compare CoT and Direct prompting under increasing amounts of S2TT data. To this end, we pseudo-label an ASR corpus by translating its transcriptions into six European languages, and train LLM-based S2TT systems with both prompting strategies at different data scales. Our results show that Direct improves more consistently as the amount of data increases, suggesting that it may become a more effective approach as larger S2TT resources are created.

Paper Structure

This paper contains 11 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Translation pseudo-labeling
  4. Speech Translation Model
  5. Experimental Setup
  6. Data
  7. Experiments
  8. Training Details
  9. Results
  10. Conclusions and Future Directions
  11. Acknowledgements

Figures (4)

  • Figure 1: Overview of the pseudo-labeling pipeline.
  • Figure 2: Prompt templates (model outputs are in bold)
  • Figure 3: Performance on Fleurs (xcomet) when scaling the amount of S2TTpl data. Results are averaged across all the considered language directions.
  • Figure 4: S2TT results evaluated on Fleurs test set with xcomet. The x-axis represents the percentage of the S2TTpl data with which the different Direct and CoT models were trained.