Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbing

William Brannon, Yogesh Virkar, Brian Thompson

TL;DR

This work delivers the first large-scale quantitative study of human dubbing, analyzing 319.57 hours across 54 titles to understand how dubbers balance semantic fidelity, naturalness, timing, and lip-sync. Using a rigorously processed English–dub corpus with cross-lingual alignments, gender and on-screen annotations, the study reveals that isochrony is frequently violated, lip-sync is a soft constraint, and isometry is an inadequate proxy for true timing alignment. It further shows substantial non-text transfer from source audio into the target dubbing, including emotion and emphasis, suggesting automatic dubbing should encode speaker traits and prosody beyond literal translation. The findings challenge common assumptions in both qualitative dubbing literature and isometry-centric automatic dubbing approaches, offering concrete directions for building more natural and faithful dubbing systems that prioritize translation quality and emotion transfer over strict duration and lip-sync constraints.

Abstract

We investigate how humans perform the task of dubbing video content from one language into another, leveraging a novel corpus of 319.57 hours of video from 54 professionally produced titles. This is the first such large-scale study we are aware of. The results challenge a number of assumptions commonly made in both qualitative literature on human dubbing and machine-learning literature on automatic dubbing, arguing for the importance of vocal naturalness and translation quality over commonly emphasized isometric (character length) and lip-sync constraints, and for a more qualified view of the importance of isochronic (timing) constraints. We also find substantial influence of the source-side audio on human dubs through channels other than the words of the translation, pointing to the need for research on ways to preserve speech characteristics, as well as semantic transfer such as emphasis/emotion, in automatic dubbing systems.

Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbing

TL;DR

This work delivers the first large-scale quantitative study of human dubbing, analyzing 319.57 hours across 54 titles to understand how dubbers balance semantic fidelity, naturalness, timing, and lip-sync. Using a rigorously processed English–dub corpus with cross-lingual alignments, gender and on-screen annotations, the study reveals that isochrony is frequently violated, lip-sync is a soft constraint, and isometry is an inadequate proxy for true timing alignment. It further shows substantial non-text transfer from source audio into the target dubbing, including emotion and emphasis, suggesting automatic dubbing should encode speaker traits and prosody beyond literal translation. The findings challenge common assumptions in both qualitative dubbing literature and isometry-centric automatic dubbing approaches, offering concrete directions for building more natural and faithful dubbing systems that prioritize translation quality and emotion transfer over strict duration and lip-sync constraints.

Abstract

We investigate how humans perform the task of dubbing video content from one language into another, leveraging a novel corpus of 319.57 hours of video from 54 professionally produced titles. This is the first such large-scale study we are aware of. The results challenge a number of assumptions commonly made in both qualitative literature on human dubbing and machine-learning literature on automatic dubbing, arguing for the importance of vocal naturalness and translation quality over commonly emphasized isometric (character length) and lip-sync constraints, and for a more qualified view of the importance of isochronic (timing) constraints. We also find substantial influence of the source-side audio on human dubs through channels other than the words of the translation, pointing to the need for research on ways to preserve speech characteristics, as well as semantic transfer such as emphasis/emotion, in automatic dubbing systems.
Paper Structure (23 sections, 3 figures, 4 tables)

This paper contains 23 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Qualitative
  4. Automatic Dubbing
  5. Empirical Studies
  6. Corpus Description & Preprocessing
  7. Segmentation and Forced Alignment
  8. Filtering
  9. Cross-lingual alignment
  10. Gender Annotations
  11. On-Screen Annotations
  12. Data Release Considerations
  13. Analysis
  14. Isochrony
  15. Isometry
  16. ...and 8 more sections

Figures (3)

  • Figure 1: Percentage change in character count from English source to human dub among onscreen lines. The plot is clipped at a 200% increase (and the character count can't decrease by more than 100%). Vertical red lines indicate a $\pm 10\%$ change.
  • Figure 2: Comet-qe scores, including means and standard deviations, broken out by onscreen/offscreen status. Note that these scores represent only data points with onscreen/offscreen annotations. No meaningful onscreen/offscreen differences are observed in comet-qe or prism-src (not shown for space) scores.
  • Figure 3: Pearson correlations of various audio properties between source and target (dubbing voice actors) within aligned word pairs ("Within-word") and within unaligned word pairs ("Across-word") in the same dialogue line. All properties show greater correlation within aligned word pairs than across them.