Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FunCineForge: A Unified Dataset Toolkit and Model for Zero-Shot Movie Dubbing in Diverse Cinematic Scenes

Jiaxuan Liu, Yang Xiang, Han Zhao, Xiangang Li, Zhenhua Ling

TL;DR

FunCineForge tackles zero-shot movie dubbing in diverse cinematic scenes by addressing data scarcity and lip-only alignment limitations with a unified pipeline and an MLLM-based dubbing model. The approach combines a large-scale CineDub-CN dataset with rich multimodal annotations and a two-part model: multimodal alignment via an MLLM and a flow-matching module that supports speaker switching. Key contributions include the first large-scale Chinese television dubbing dataset with correction protocols, a multimodal CoT-driven correction mechanism, and a flow-based dubbing model that delivers precise lip sync, natural timbre transfer, and expressive emotion across monologue, narration, dialogue, and multi-speaker scenes. The results demonstrate consistent gains over SOTA baselines and highlight the practical impact for scalable, high-quality film and television dubbing, with potential for multilingual extension and joint video-to-audio generation.

Abstract

Movie dubbing is the task of synthesizing speech from scripts conditioned on video scenes, requiring accurate lip sync, faithful timbre transfer, and proper modeling of character identity and emotion. However, existing methods face two major limitations: (1) high-quality multimodal dubbing datasets are limited in scale, suffer from high word error rates, contain sparse annotations, rely on costly manual labeling, and are restricted to monologue scenes, all of which hinder effective model training; (2) existing dubbing models rely solely on the lip region to learn audio-visual alignment, which limits their applicability to complex live-action cinematic scenes, and exhibit suboptimal performance in lip sync, speech quality, and emotional expressiveness. To address these issues, we propose FunCineForge, which comprises an end-to-end production pipeline for large-scale dubbing datasets and an MLLM-based dubbing model designed for diverse cinematic scenes. Using the pipeline, we construct the first Chinese television dubbing dataset with rich annotations, and demonstrate the high quality of these data. Experiments across monologue, narration, dialogue, and multi-speaker scenes show that our dubbing model consistently outperforms SOTA methods in audio quality, lip sync, timbre transfer, and instruction following. Code and demos are available at https://anonymous.4open.science/w/FunCineForge.

FunCineForge: A Unified Dataset Toolkit and Model for Zero-Shot Movie Dubbing in Diverse Cinematic Scenes

TL;DR

FunCineForge tackles zero-shot movie dubbing in diverse cinematic scenes by addressing data scarcity and lip-only alignment limitations with a unified pipeline and an MLLM-based dubbing model. The approach combines a large-scale CineDub-CN dataset with rich multimodal annotations and a two-part model: multimodal alignment via an MLLM and a flow-matching module that supports speaker switching. Key contributions include the first large-scale Chinese television dubbing dataset with correction protocols, a multimodal CoT-driven correction mechanism, and a flow-based dubbing model that delivers precise lip sync, natural timbre transfer, and expressive emotion across monologue, narration, dialogue, and multi-speaker scenes. The results demonstrate consistent gains over SOTA baselines and highlight the practical impact for scalable, high-quality film and television dubbing, with potential for multilingual extension and joint video-to-audio generation.

Abstract

Movie dubbing is the task of synthesizing speech from scripts conditioned on video scenes, requiring accurate lip sync, faithful timbre transfer, and proper modeling of character identity and emotion. However, existing methods face two major limitations: (1) high-quality multimodal dubbing datasets are limited in scale, suffer from high word error rates, contain sparse annotations, rely on costly manual labeling, and are restricted to monologue scenes, all of which hinder effective model training; (2) existing dubbing models rely solely on the lip region to learn audio-visual alignment, which limits their applicability to complex live-action cinematic scenes, and exhibit suboptimal performance in lip sync, speech quality, and emotional expressiveness. To address these issues, we propose FunCineForge, which comprises an end-to-end production pipeline for large-scale dubbing datasets and an MLLM-based dubbing model designed for diverse cinematic scenes. Using the pipeline, we construct the first Chinese television dubbing dataset with rich annotations, and demonstrate the high quality of these data. Experiments across monologue, narration, dialogue, and multi-speaker scenes show that our dubbing model consistently outperforms SOTA methods in audio quality, lip sync, timbre transfer, and instruction following. Code and demos are available at https://anonymous.4open.science/w/FunCineForge.
Paper Structure (31 sections, 6 equations, 3 figures, 2 tables)

This paper contains 31 sections, 6 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Dubbing Datasets
  4. LLM-based Text-to-Speech
  5. Speaker Diarization for Cinematic Scenes
  6. FunCineForge
  7. Dataset Pipeline
  8. Video Standardization and Segmentation
  9. Vocal Separation and Filtering
  10. Audio-Visual Speaker Diarization
  11. Multimodal CoT Correction with MLLM
  12. Dubbing Model
  13. MLLM with Multimodal Alignment
  14. Flow Matching with Speaker Switching
  15. Experiments
  16. ...and 16 more sections

Figures (3)

  • Figure 1: The overview of FunCineForge. The dataset pipeline automatically transforms raw film and television sources into structured multimodal data, which are used to train and evaluate the dubbing model. During inference, given a silent video clip, dubbing script, clue instructions, scene category, and reference speeches, the model synthesizes scene-consistent speech.
  • Figure 2: The main architecture of FunCineForge. The framework consists of a dataset pipeline and a dubbing model. The dubbing model comprises two components: an MLLM for multimodal alignment, and a flow matching module with speaker switching.
  • Figure 3: The left subfigure shows the distributions of scene categories, speaker age groups, and gender, while the right subfigure presents a timbre-related keyword cloud of the CineDub-CN dataset.