Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Film-Making Production Dialogue, Narration, Monologue Adaptive Moving Dubbing Benchmarks

Chaoyi Wang, Junjie Zheng, Zihao Chen, Shiyu Xia, Chaofan Ding, Xiaohao Zhang, Xi Tao, Xiaoming He, Xinhan Di

TL;DR

The paper tackles the gap in evaluating movie dubbing under real-world production constraints by introducing TA-Dubbing, a comprehensive benchmark that jointly assesses comprehension of movie clips and the quality of adaptive dubbing across dialogue, narration, monologue, and actor attributes. It provides a multimodal CoT-based dataset (about 140k clips with 130k training and 10k testing) and a structured evaluation suite covering recognition, speech quality, and actor-attribute metrics, complemented by open-source access and a leaderboard. Key contributions include the dataset with step-by-step CoT reasoning for scene-type classification, the full metric toolkit, and a public platform to drive progression in film dubbing models. Experimental results on current state-of-the-art models reveal gaps in adaptive dubbing and scene-type recognition, highlighting the need for further research to enable production-ready, actor-aware dubbing systems.

Abstract

Movie dubbing has advanced significantly, yet assessing the real-world effectiveness of these models remains challenging. A comprehensive evaluation benchmark is crucial for two key reasons: 1) Existing metrics fail to fully capture the complexities of dialogue, narration, monologue, and actor adaptability in movie dubbing. 2) A practical evaluation system should offer valuable insights to improve movie dubbing quality and advancement in film production. To this end, we introduce Talking Adaptive Dubbing Benchmarks (TA-Dubbing), designed to improve film production by adapting to dialogue, narration, monologue, and actors in movie dubbing. TA-Dubbing offers several key advantages: 1) Comprehensive Dimensions: TA-Dubbing covers a variety of dimensions of movie dubbing, incorporating metric evaluations for both movie understanding and speech generation. 2) Versatile Benchmarking: TA-Dubbing is designed to evaluate state-of-the-art movie dubbing models and advanced multi-modal large language models. 3) Full Open-Sourcing: We fully open-source TA-Dubbing at https://github.com/woka- 0a/DeepDubber- V1 including all video suits, evaluation methods, annotations. We also continuously integrate new movie dubbing models into the TA-Dubbing leaderboard at https://github.com/woka- 0a/DeepDubber-V1 to drive forward the field of movie dubbing.

Towards Film-Making Production Dialogue, Narration, Monologue Adaptive Moving Dubbing Benchmarks

TL;DR

The paper tackles the gap in evaluating movie dubbing under real-world production constraints by introducing TA-Dubbing, a comprehensive benchmark that jointly assesses comprehension of movie clips and the quality of adaptive dubbing across dialogue, narration, monologue, and actor attributes. It provides a multimodal CoT-based dataset (about 140k clips with 130k training and 10k testing) and a structured evaluation suite covering recognition, speech quality, and actor-attribute metrics, complemented by open-source access and a leaderboard. Key contributions include the dataset with step-by-step CoT reasoning for scene-type classification, the full metric toolkit, and a public platform to drive progression in film dubbing models. Experimental results on current state-of-the-art models reveal gaps in adaptive dubbing and scene-type recognition, highlighting the need for further research to enable production-ready, actor-aware dubbing systems.

Abstract

Movie dubbing has advanced significantly, yet assessing the real-world effectiveness of these models remains challenging. A comprehensive evaluation benchmark is crucial for two key reasons: 1) Existing metrics fail to fully capture the complexities of dialogue, narration, monologue, and actor adaptability in movie dubbing. 2) A practical evaluation system should offer valuable insights to improve movie dubbing quality and advancement in film production. To this end, we introduce Talking Adaptive Dubbing Benchmarks (TA-Dubbing), designed to improve film production by adapting to dialogue, narration, monologue, and actors in movie dubbing. TA-Dubbing offers several key advantages: 1) Comprehensive Dimensions: TA-Dubbing covers a variety of dimensions of movie dubbing, incorporating metric evaluations for both movie understanding and speech generation. 2) Versatile Benchmarking: TA-Dubbing is designed to evaluate state-of-the-art movie dubbing models and advanced multi-modal large language models. 3) Full Open-Sourcing: We fully open-source TA-Dubbing at https://github.com/woka- 0a/DeepDubber- V1 including all video suits, evaluation methods, annotations. We also continuously integrate new movie dubbing models into the TA-Dubbing leaderboard at https://github.com/woka- 0a/DeepDubber-V1 to drive forward the field of movie dubbing.
Paper Structure (12 sections, 3 figures, 1 table)

This paper contains 12 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Visual Voice Cloning
  4. Evaluation of Movie Dubbing Models
  5. Benchmark Suite
  6. Movie Dubbing Dataset Suite
  7. Evaluation Metrics Suite
  8. 1. Dialogue, Narration, Monologue Recognition Evaluation Metrics
  9. 2. Speech Quality Evaluation Metrics
  10. 3. Actor Recognition Evaluation Metrics
  11. Experiments
  12. Discussion

Figures (3)

  • Figure 1: Proposed dataset with multi-type annotations, including annotation for lips, faces, scene-type, actor name, actor gender, actor age, voice emotion and chain-of-thoughts annotations for dialogue, narration and monologue.
  • Figure 2: The reasoning stages of movie scene type(dialogue, narration, monologue) CoT annotations.
  • Figure 3: Visualization of generated movie dubbing samples via state-of-the-art models. The green rectangles highlight key regions that have significant differences in overall expressiveness.