Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?

Sara Papi, Peter Polak, Ondřej Bojar, Dominik Macháček

TL;DR

This paper analyzes the state of simultaneous speech-to-text translation (SimulST) and reveals that the field largely relies on gold, bounded, pre-segmented speech and suffers from terminological chaos. It formalizes SimulST as a six-step process, introduces unified terminology and a taxonomy, and surveys 110 papers to identify trends, gaps, and best practices. The authors advocate automatic pre-segmentation, explicit input-type reporting, and the development of unbounded-speech evaluation frameworks, as well as contextual and user-centered evaluation approaches. Overall, the work aims to steer the field toward realistic, end-to-end systems capable of processing continuous audio streams with latency-sensitive, user-focused translations.

Abstract

Simultaneous speech-to-text translation (SimulST) translates source-language speech into target-language text concurrently with the speaker's speech, ensuring low latency for better user comprehension. Despite its intended application to unbounded speech, most research has focused on human pre-segmented speech, simplifying the task and overlooking significant challenges. This narrow focus, coupled with widespread terminological inconsistencies, is limiting the applicability of research outcomes to real-world applications, ultimately hindering progress in the field. Our extensive literature review of 110 papers not only reveals these critical issues in current research but also serves as the foundation for our key contributions. We 1) define the steps and core components of a SimulST system, proposing a standardized terminology and taxonomy; 2) conduct a thorough analysis of community trends, and 3) offer concrete recommendations and future directions to bridge the gaps in existing literature, from evaluation frameworks to system architectures, for advancing the field towards more realistic and effective SimulST solutions.

How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?

TL;DR

This paper analyzes the state of simultaneous speech-to-text translation (SimulST) and reveals that the field largely relies on gold, bounded, pre-segmented speech and suffers from terminological chaos. It formalizes SimulST as a six-step process, introduces unified terminology and a taxonomy, and surveys 110 papers to identify trends, gaps, and best practices. The authors advocate automatic pre-segmentation, explicit input-type reporting, and the development of unbounded-speech evaluation frameworks, as well as contextual and user-centered evaluation approaches. Overall, the work aims to steer the field toward realistic, end-to-end systems capable of processing continuous audio streams with latency-sensitive, user-focused translations.

Abstract

Simultaneous speech-to-text translation (SimulST) translates source-language speech into target-language text concurrently with the speaker's speech, ensuring low latency for better user comprehension. Despite its intended application to unbounded speech, most research has focused on human pre-segmented speech, simplifying the task and overlooking significant challenges. This narrow focus, coupled with widespread terminological inconsistencies, is limiting the applicability of research outcomes to real-world applications, ultimately hindering progress in the field. Our extensive literature review of 110 papers not only reveals these critical issues in current research but also serves as the foundation for our key contributions. We 1) define the steps and core components of a SimulST system, proposing a standardized terminology and taxonomy; 2) conduct a thorough analysis of community trends, and 3) offer concrete recommendations and future directions to bridge the gaps in existing literature, from evaluation frameworks to system architectures, for advancing the field towards more realistic and effective SimulST solutions.

Paper Structure

This paper contains 44 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Offline Speech Translation
  4. Audio Segmentation
  5. Long-Form Speech
  6. What is Simultaneous Speech-to-Text Translation?
  7. Process Decomposition
  8. Terminology and Models' Components
  9. Bounded vs. Unbounded Input Speech.
  10. Direct vs. Cascade Architecture.
  11. Incremental vs. Re-translation.
  12. Computationally aware vs. unaware latency.
  13. Is it "Real" Simultaneous Translation?
  14. The Terminological Chaos.
  15. Humans will not segment our audio.
  16. ...and 29 more sections

Figures (3)

  • Figure 2: Taxonomy of the SimulST solutions.
  • Figure 3: Waffle plot of the term "simultaneous" and commonly used synonyms ("streaming", "real-time", and "online") among the 110 categorized papers.
  • Figure 4: Number of papers in our survey employing direct or cascade simultaneous ST architectures throughout the years. 2024* means that the data are incomplete since the year is not finished yet.