Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating the IWSLT2023 Speech Translation Tasks: Human Annotations, Automatic Metrics, and Segmentation

Matthias Sperber, Ondřej Bojar, Barry Haddow, Dávid Javorský, Xutai Ma, Matteo Negri, Jan Niehues, Peter Polák, Elizabeth Salesky, Katsuhito Sudoh, Marco Turchi

TL;DR

This paper tackles the challenge of evaluating speech translation (ST) with trustworthy human judgments, addressing segmentation noise and real-time constraints. It proposes a pragmatic evaluation pipeline that combines automatic resegmentation of outputs with direct assessment using segment context, and corroborates these human judgments with MQM and Continuous Rating. Empirically, direct assessment correlates highly with automatic metric scores, and the trainable COMET metric generally outperforms chrF in ST settings, even under segmentation noise, though correlations vary by task and data size. The authors also publish the collected human annotations to spur further research and cross-modality evaluation in speech translation, highlighting practical implications for designing robust ST evaluation protocols.

Abstract

Human evaluation is a critical component in machine translation system development and has received much attention in text translation research. However, little prior work exists on the topic of human evaluation for speech translation, which adds additional challenges such as noisy data and segmentation mismatches. We take first steps to fill this gap by conducting a comprehensive human evaluation of the results of several shared tasks from the last International Workshop on Spoken Language Translation (IWSLT 2023). We propose an effective evaluation strategy based on automatic resegmentation and direct assessment with segment context. Our analysis revealed that: 1) the proposed evaluation strategy is robust and scores well-correlated with other types of human judgements; 2) automatic metrics are usually, but not always, well-correlated with direct assessment scores; and 3) COMET as a slightly stronger automatic metric than chrF, despite the segmentation noise introduced by the resegmentation step systems. We release the collected human-annotated data in order to encourage further investigation.

Evaluating the IWSLT2023 Speech Translation Tasks: Human Annotations, Automatic Metrics, and Segmentation

TL;DR

This paper tackles the challenge of evaluating speech translation (ST) with trustworthy human judgments, addressing segmentation noise and real-time constraints. It proposes a pragmatic evaluation pipeline that combines automatic resegmentation of outputs with direct assessment using segment context, and corroborates these human judgments with MQM and Continuous Rating. Empirically, direct assessment correlates highly with automatic metric scores, and the trainable COMET metric generally outperforms chrF in ST settings, even under segmentation noise, though correlations vary by task and data size. The authors also publish the collected human annotations to spur further research and cross-modality evaluation in speech translation, highlighting practical implications for designing robust ST evaluation protocols.

Abstract

Human evaluation is a critical component in machine translation system development and has received much attention in text translation research. However, little prior work exists on the topic of human evaluation for speech translation, which adds additional challenges such as noisy data and segmentation mismatches. We take first steps to fill this gap by conducting a comprehensive human evaluation of the results of several shared tasks from the last International Workshop on Spoken Language Translation (IWSLT 2023). We propose an effective evaluation strategy based on automatic resegmentation and direct assessment with segment context. Our analysis revealed that: 1) the proposed evaluation strategy is robust and scores well-correlated with other types of human judgements; 2) automatic metrics are usually, but not always, well-correlated with direct assessment scores; and 3) COMET as a slightly stronger automatic metric than chrF, despite the segmentation noise introduced by the resegmentation step systems. We release the collected human-annotated data in order to encourage further investigation.
Paper Structure (26 sections, 2 figures, 10 tables)

This paper contains 26 sections, 2 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Task Description
  3. Conditions
  4. Offline
  5. Multilingual
  6. Simultaneous
  7. Test Data
  8. TED
  9. ACL
  10. Evaluation
  11. Resegmentation
  12. Automatic Metrics
  13. Human Evaluation
  14. Direct assessment
  15. MQM
  16. ...and 11 more sections

Figures (2)

  • Figure 1: To parallelize system outputs with reference translations, we re-segment hypotheses by minimizing the overall WER to the reference with mwerSegmenter, which supports many-to-many realignment.
  • Figure 2: Example for annotation of re-segmented English-German system outputs. Gloss is provided in italics. The human annotator, if looking only at the re-segmented system output of the current sentence, would find the main verb missing, the beginning of the sentence not capitalized, and the sentences ending in a comma instead of full stop, and penalize the sentence accordingly. As a remedy, we also present system outputs for adjacent segments to the annotator. The main verb (gehen) is located in the system output for the previous sentence, and by being presented with the context, the annotator will be able to see that this translation has no grammatical issues nor problems with casing or punctuation.