Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

StyleDubber: Towards Multi-Scale Style Learning for Movie Dubbing

Gaoxiang Cong, Yuankai Qi, Liang Li, Amin Beheshti, Zhedong Zhang, Anton van den Hengel, Ming-Hsuan Yang, Chenggang Yan, Qingming Huang

TL;DR

StyleDubber tackles Visual Voice Cloning (V2C) by moving from frame-level to phoneme- and utterance-level style learning, combining a Multimodal Phoneme-level Adaptor (MPA), Phoneme-guided Lip Aligner (PLA), and Utterance-level Style Learning (USL) to generate temporally aligned, emotionally expressive, and identity-consistent dubbing. It learns pronunciation style from reference audio and emotion from video, then refines overall timbre at the utterance level while preserving lip-sync via a dedicated lip-aligner and duration predictor. Extensive experiments on V2C-Animation and GRID show state-of-the-art results across pronunciation accuracy, speaker similarity, and timing alignment, with strong generalization to unseen speakers. The approach offers practical impact for film dubbing and AI-assisted audio-visual content creation by producing natural, synchronized, and style-consistent dubbed speech.

Abstract

Given a script, the challenge in Movie Dubbing (Visual Voice Cloning, V2C) is to generate speech that aligns well with the video in both time and emotion, based on the tone of a reference audio track. Existing state-of-the-art V2C models break the phonemes in the script according to the divisions between video frames, which solves the temporal alignment problem but leads to incomplete phoneme pronunciation and poor identity stability. To address this problem, we propose StyleDubber, which switches dubbing learning from the frame level to phoneme level. It contains three main components: (1) A multimodal style adaptor operating at the phoneme level to learn pronunciation style from the reference audio, and generate intermediate representations informed by the facial emotion presented in the video; (2) An utterance-level style learning module, which guides both the mel-spectrogram decoding and the refining processes from the intermediate embeddings to improve the overall style expression; And (3) a phoneme-guided lip aligner to maintain lip sync. Extensive experiments on two of the primary benchmarks, V2C and Grid, demonstrate the favorable performance of the proposed method as compared to the current stateof-the-art. The code will be made available at https://github.com/GalaxyCong/StyleDubber.

StyleDubber: Towards Multi-Scale Style Learning for Movie Dubbing

TL;DR

StyleDubber tackles Visual Voice Cloning (V2C) by moving from frame-level to phoneme- and utterance-level style learning, combining a Multimodal Phoneme-level Adaptor (MPA), Phoneme-guided Lip Aligner (PLA), and Utterance-level Style Learning (USL) to generate temporally aligned, emotionally expressive, and identity-consistent dubbing. It learns pronunciation style from reference audio and emotion from video, then refines overall timbre at the utterance level while preserving lip-sync via a dedicated lip-aligner and duration predictor. Extensive experiments on V2C-Animation and GRID show state-of-the-art results across pronunciation accuracy, speaker similarity, and timing alignment, with strong generalization to unseen speakers. The approach offers practical impact for film dubbing and AI-assisted audio-visual content creation by producing natural, synchronized, and style-consistent dubbed speech.

Abstract

Given a script, the challenge in Movie Dubbing (Visual Voice Cloning, V2C) is to generate speech that aligns well with the video in both time and emotion, based on the tone of a reference audio track. Existing state-of-the-art V2C models break the phonemes in the script according to the divisions between video frames, which solves the temporal alignment problem but leads to incomplete phoneme pronunciation and poor identity stability. To address this problem, we propose StyleDubber, which switches dubbing learning from the frame level to phoneme level. It contains three main components: (1) A multimodal style adaptor operating at the phoneme level to learn pronunciation style from the reference audio, and generate intermediate representations informed by the facial emotion presented in the video; (2) An utterance-level style learning module, which guides both the mel-spectrogram decoding and the refining processes from the intermediate embeddings to improve the overall style expression; And (3) a phoneme-guided lip aligner to maintain lip sync. Extensive experiments on two of the primary benchmarks, V2C and Grid, demonstrate the favorable performance of the proposed method as compared to the current stateof-the-art. The code will be made available at https://github.com/GalaxyCong/StyleDubber.
Paper Structure (22 sections, 16 equations, 4 figures, 8 tables)

This paper contains 22 sections, 16 equations, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Method
  4. Overview
  5. Multimodal Phoneme-level Adaptor
  6. Phoneme-guided Lip Aligner
  7. Utterance-level Style Learning
  8. Training
  9. Experiments
  10. Implementation Details
  11. Dataset
  12. Evaluation Metrics
  13. Performance Evaluations
  14. Qualitative Results
  15. Ablation Studies
  16. ...and 7 more sections

Figures (4)

  • Figure 1: (a) Illustration of the V2C task. (b) Our StyleDubber learns speech styles on two levels: phoneme-level focuses on pronunciation details, while utterance-level emphasizes the overall consistency like timbre.
  • Figure 2: The main architecture of the proposed StyleDubber. It consists of a) Multimodal Phoneme-level Adaptor (MPA) (Sec. \ref{['sec:mpa']}), b) Phoneme-guided Lip Aligner (PLA) (Sec. \ref{['sec:pla']}), and c) Utterance-level Style Learning (USL) (Sec. \ref{['sec:usl']}). Note that $\oplus$ is intended to denote vector addition.
  • Figure 3: Mel-spectrograms of four synthesized audio samples under the Dub 2.0 setting. The green and blue rectangles highlight key regions that have significant differences in reconstruction details and duration pause.
  • Figure 4: V2C dataset is more challenging than TTS-baseline datasets: (a) fewer samples (only 6567 for training), (b) shorter duration (mostly smaller than 5s), (c) greater variance (pitch), (d) more noise (background sound and music).