Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MM-MovieDubber: Towards Multi-Modal Learning for Multi-Modal Movie Dubbing

Junjie Zheng, Zihao Chen, Chaofan Ding, Yunming Liang, Yihan Fan, Huan Yang, Lei Xie, Xinhan Di

TL;DR

This work tackles the movie dubbing problem by introducing MM-MovieDubber, a two-stage, multi-modal framework that first uses a multi-modal large language model to extract a video-informed understanding for dubbing and then generates contextually appropriate, lip-synced speech via a diffusion-based generator conditioned on visual, textual, and inferred attributes. A new movie-dubbing dataset with annotations for dubbing types and fine-grained attributes supports learning and evaluation. The method integrates cross-modal cues through a ControlNet-style conditioning mechanism, including duration constraints and classifier-free guidance, achieving state-of-the-art results on V2C-Animation and GRID benchmarks and demonstrating robustness in zero-shot and fine-grained scenarios. The approach reduces WER and MCD while improving SPK-SIM and EMO-SIM, indicating stronger pronunciation, timbre, and emotional alignment with on-screen actions, thereby enhancing practical dubbing quality and synchronization.

Abstract

Current movie dubbing technology can produce the desired speech using a reference voice and input video, maintaining perfect synchronization with the visuals while effectively conveying the intended emotions. However, crucial aspects of movie dubbing, including adaptation to various dubbing styles, effective handling of dialogue, narration, and monologues, as well as consideration of subtle details such as speaker age and gender, remain insufficiently explored. To tackle these challenges, we introduce a multi-modal generative framework. First, it utilizes a multi-modal large vision-language model (VLM) to analyze visual inputs, enabling the recognition of dubbing types and fine-grained attributes. Second, it produces high-quality dubbing using large speech generation models, guided by multi-modal inputs. Additionally, a movie dubbing dataset with annotations for dubbing types and subtle details is constructed to enhance movie understanding and improve dubbing quality for the proposed multi-modal framework. Experimental results across multiple benchmark datasets show superior performance compared to state-of-the-art (SOTA) methods. In details, the LSE-D, SPK-SIM, EMO-SIM, and MCD exhibit improvements of up to 1.09%, 8.80%, 19.08%, and 18.74%, respectively.

MM-MovieDubber: Towards Multi-Modal Learning for Multi-Modal Movie Dubbing

TL;DR

This work tackles the movie dubbing problem by introducing MM-MovieDubber, a two-stage, multi-modal framework that first uses a multi-modal large language model to extract a video-informed understanding for dubbing and then generates contextually appropriate, lip-synced speech via a diffusion-based generator conditioned on visual, textual, and inferred attributes. A new movie-dubbing dataset with annotations for dubbing types and fine-grained attributes supports learning and evaluation. The method integrates cross-modal cues through a ControlNet-style conditioning mechanism, including duration constraints and classifier-free guidance, achieving state-of-the-art results on V2C-Animation and GRID benchmarks and demonstrating robustness in zero-shot and fine-grained scenarios. The approach reduces WER and MCD while improving SPK-SIM and EMO-SIM, indicating stronger pronunciation, timbre, and emotional alignment with on-screen actions, thereby enhancing practical dubbing quality and synchronization.

Abstract

Current movie dubbing technology can produce the desired speech using a reference voice and input video, maintaining perfect synchronization with the visuals while effectively conveying the intended emotions. However, crucial aspects of movie dubbing, including adaptation to various dubbing styles, effective handling of dialogue, narration, and monologues, as well as consideration of subtle details such as speaker age and gender, remain insufficiently explored. To tackle these challenges, we introduce a multi-modal generative framework. First, it utilizes a multi-modal large vision-language model (VLM) to analyze visual inputs, enabling the recognition of dubbing types and fine-grained attributes. Second, it produces high-quality dubbing using large speech generation models, guided by multi-modal inputs. Additionally, a movie dubbing dataset with annotations for dubbing types and subtle details is constructed to enhance movie understanding and improve dubbing quality for the proposed multi-modal framework. Experimental results across multiple benchmark datasets show superior performance compared to state-of-the-art (SOTA) methods. In details, the LSE-D, SPK-SIM, EMO-SIM, and MCD exhibit improvements of up to 1.09%, 8.80%, 19.08%, and 18.74%, respectively.

Paper Structure

This paper contains 14 sections, 9 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Overview
  4. Stage 1: Multi-modal Large Language Model (mLLM) Learning
  5. Stage 2: Mutli-Conditioned Speech Generation
  6. Stage 2.1: Speech Generation Foundation Pre-Training
  7. Stage 2.2: Speech Generation With Multi-Modal Conditions Tuning
  8. Experiments
  9. Datasets
  10. Implementation Details
  11. Evaluation
  12. Results
  13. Comparison with SOTA
  14. Conclusion

Figures (3)

  • Figure 1: Curent Dubbing models congLearningDubMovies2023cong2024styledubbermultiscalestylelearningzhang2024from (Left). Proposed Dubbing Models (Right) For dubbing types and fine-grained attributes.
  • Figure 2: MM-MovieDubber pipeline with multi-stage, multi-modal training.
  • Figure 3: Proposed dataset with multi-type annotations, including annotation for lips, faces, scene-type, speaker gender, speaker age, voice emotion.