Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Case Study on Contextual Machine Translation in a Professional Scenario of Subtitling

Sebastian Vincent, Charlotte Prescott, Chris Bayliss, Chris Oakley, Carolina Scarton

TL;DR

The study investigates whether incorporating extra-textual context into MT improves subtitle translation in a professional, multi-modal subtitling workflow. It compares a context-aware MTCue model against non-contextual MT and human references across two language pairs, using both automatic metrics and a post-editing trial to measure quality and editing effort. Results show that contextual MT can reduce context-related and stylistic errors, particularly in English–French, while maintaining or improving overall translation quality, and that post-editing with MT outputs dramatically lowers effort compared to translating from scratch. These findings support the continued development of fully contextual MT for industry use, with emphasis on understanding which contextual signals yield the greatest gains and how to optimize post-editing workflows.

Abstract

Incorporating extra-textual context such as film metadata into the machine translation (MT) pipeline can enhance translation quality, as indicated by automatic evaluation in recent work. However, the positive impact of such systems in industry remains unproven. We report on an industrial case study carried out to investigate the benefit of MT in a professional scenario of translating TV subtitles with a focus on how leveraging extra-textual context impacts post-editing. We found that post-editors marked significantly fewer context-related errors when correcting the outputs of MTCue, the context-aware model, as opposed to non-contextual models. We also present the results of a survey of the employed post-editors, which highlights contextual inadequacy as a significant gap consistently observed in MT. Our findings strengthen the motivation for further work within fully contextual MT.

A Case Study on Contextual Machine Translation in a Professional Scenario of Subtitling

TL;DR

The study investigates whether incorporating extra-textual context into MT improves subtitle translation in a professional, multi-modal subtitling workflow. It compares a context-aware MTCue model against non-contextual MT and human references across two language pairs, using both automatic metrics and a post-editing trial to measure quality and editing effort. Results show that contextual MT can reduce context-related and stylistic errors, particularly in English–French, while maintaining or improving overall translation quality, and that post-editing with MT outputs dramatically lowers effort compared to translating from scratch. These findings support the continued development of fully contextual MT for industry use, with emphasis on understanding which contextual signals yield the greatest gains and how to optimize post-editing workflows.

Abstract

Incorporating extra-textual context such as film metadata into the machine translation (MT) pipeline can enhance translation quality, as indicated by automatic evaluation in recent work. However, the positive impact of such systems in industry remains unproven. We report on an industrial case study carried out to investigate the benefit of MT in a professional scenario of translating TV subtitles with a focus on how leveraging extra-textual context impacts post-editing. We found that post-editors marked significantly fewer context-related errors when correcting the outputs of MTCue, the context-aware model, as opposed to non-contextual models. We also present the results of a survey of the employed post-editors, which highlights contextual inadequacy as a significant gap consistently observed in MT. Our findings strengthen the motivation for further work within fully contextual MT.
Paper Structure (20 sections, 2 equations, 4 figures, 5 tables)

This paper contains 20 sections, 2 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Experimental Setup
  4. Automatic evaluation
  5. Post-Editing Setup and Metrics
  6. Worker setup
  7. Details regarding the PEs
  8. Results of Automatic Evaluation
  9. Results of the Post-Editing Study
  10. Error Analysis
  11. Error post-processing
  12. Results
  13. Analysis of Effort and Quality
  14. Effort per PE
  15. Approach to Ref
  16. ...and 5 more sections

Figures (4)

  • Figure 1: A compressed snapshot of ZOOSubs.
  • Figure 2: BLEU, Comet and PMI scores obtained by the evaluated models. Asterisks (*) over bars indicate the best result along with all statistically indistinguishable results computed either via bootstrap resampling (or t-test for PMI), $p=0.05$.
  • Figure 3: Effort for each PE within both language pairs.
  • Figure 4: Effort comparison of FST and post-editing MT.