Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Continuous Rating as Reliable Human Evaluation of Simultaneous Speech Translation

Dávid Javorský, Dominik Macháček, Ondřej Bojar

TL;DR

This study indicates users’ preferences on subtitle layout and presentation style and provides a significant evidence that users with advanced source language knowledge prefer low latency over fewer re-translations.

Abstract

Simultaneous speech translation (SST) can be evaluated on simulated online events where human evaluators watch subtitled videos and continuously express their satisfaction by pressing buttons (so called Continuous Rating). Continuous Rating is easy to collect, but little is known about its reliability, or relation to comprehension of foreign language document by SST users. In this paper, we contrast Continuous Rating with factual questionnaires on judges with different levels of source language knowledge. Our results show that Continuous Rating is easy and reliable SST quality assessment if the judges have at least limited knowledge of the source language. Our study indicates users' preferences on subtitle layout and presentation style and, most importantly, provides a significant evidence that users with advanced source language knowledge prefer low latency over fewer re-translations.

Continuous Rating as Reliable Human Evaluation of Simultaneous Speech Translation

TL;DR

This study indicates users’ preferences on subtitle layout and presentation style and provides a significant evidence that users with advanced source language knowledge prefer low latency over fewer re-translations.

Abstract

Simultaneous speech translation (SST) can be evaluated on simulated online events where human evaluators watch subtitled videos and continuously express their satisfaction by pressing buttons (so called Continuous Rating). Continuous Rating is easy to collect, but little is known about its reliability, or relation to comprehension of foreign language document by SST users. In this paper, we contrast Continuous Rating with factual questionnaires on judges with different levels of source language knowledge. Our results show that Continuous Rating is easy and reliable SST quality assessment if the judges have at least limited knowledge of the source language. Our study indicates users' preferences on subtitle layout and presentation style and, most importantly, provides a significant evidence that users with advanced source language knowledge prefer low latency over fewer re-translations.
Paper Structure (26 sections, 3 figures, 8 tables)

This paper contains 26 sections, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Our Contributions
  3. Related Work
  4. Evaluation Campaign
  5. Translation System
  6. Selection of Documents
  7. Subtitler: Subtitle Presentation
  8. Web Application as Simulation Environment
  9. Types of Feedback
  10. Continuous Rating
  11. Questionnaires
  12. Judges
  13. Results
  14. Comprehension Levels
  15. Layout Preference
  16. ...and 11 more sections

Figures (3)

  • Figure 1: A detail of the default layout with the video document "Dinge Erklärt: Impfen...". The video is at the top, overlaid by two lines of subtitles in Czech, followed by buttons for Continuous Rating. The button labels are: 1: Worse; 2: Average; 3: Good; 0: I do not understand at all.
  • Figure 2: The average count of answers per judge for each proficiency level. Top: Correct (OK/OK-, blue bars) and incorrect (wrong/unknown, orange bars) answers vs Continuous Rating at the time when the answer was disclosed in the original document (x-axis, 0 means worst, 3 the best), distributed by source language proficiency level of the judges. Bottom: From which source the judges learned the correct or partially correct answer; subtitles in red, sound in yellow.
  • Figure 3: The average count of answers per judge for each proficiency level. Left: OK, OK-, wrong, unknown and forgotten answers vs Continuous Rating at the time when the answer was disclosed in the original document (x-axis, 0 means worst, 3 the best), distributed by source language proficiency level of the judges: from zero through beginners (A1, A2) and intermediate (B1, B2) to advanced (C1, C2). Middle: From which source the judges learned the correct (OK) or partially correct (OK-) answer. Right: From which source the judges learned all answers, regardless of their evaluation.