Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

JoyGen: Audio-Driven 3D Depth-Aware Talking-Face Video Editing

Qili Wang, Dajiang Wu, Zihang Xu, Junshi Huang, Jun Lv

TL;DR

JoyGen addresses the challenge of precise lip-audio synchronization and high visual fidelity in editing talking-face videos. It introduces a two-stage approach that first derives audio-driven lip motion using a 3DMM-based identity plus an expression predictor, then fuses audio features with mouth-depth maps to supervise high-quality, synchronized facial synthesis via a single-step latent UNet. The method is supported by a new 130-hour Chinese talking-face dataset and evaluation on the HDTF dataset, showing state-of-the-art lip-sync metrics and visual quality. This depth-aware, audio-guided framework enables more accurate lip edits in existing videos, with practical implications for multilingual facial video editing and synthetic media generation.

Abstract

Significant progress has been made in talking-face video generation research; however, precise lip-audio synchronization and high visual quality remain challenging in editing lip shapes based on input audio. This paper introduces JoyGen, a novel two-stage framework for talking-face generation, comprising audio-driven lip motion generation and visual appearance synthesis. In the first stage, a 3D reconstruction model and an audio2motion model predict identity and expression coefficients respectively. Next, by integrating audio features with a facial depth map, we provide comprehensive supervision for precise lip-audio synchronization in facial generation. Additionally, we constructed a Chinese talking-face dataset containing 130 hours of high-quality video. JoyGen is trained on the open-source HDTF dataset and our curated dataset. Experimental results demonstrate superior lip-audio synchronization and visual quality achieved by our method.

JoyGen: Audio-Driven 3D Depth-Aware Talking-Face Video Editing

TL;DR

JoyGen addresses the challenge of precise lip-audio synchronization and high visual fidelity in editing talking-face videos. It introduces a two-stage approach that first derives audio-driven lip motion using a 3DMM-based identity plus an expression predictor, then fuses audio features with mouth-depth maps to supervise high-quality, synchronized facial synthesis via a single-step latent UNet. The method is supported by a new 130-hour Chinese talking-face dataset and evaluation on the HDTF dataset, showing state-of-the-art lip-sync metrics and visual quality. This depth-aware, audio-guided framework enables more accurate lip edits in existing videos, with practical implications for multilingual facial video editing and synthetic media generation.

Abstract

Significant progress has been made in talking-face video generation research; however, precise lip-audio synchronization and high visual quality remain challenging in editing lip shapes based on input audio. This paper introduces JoyGen, a novel two-stage framework for talking-face generation, comprising audio-driven lip motion generation and visual appearance synthesis. In the first stage, a 3D reconstruction model and an audio2motion model predict identity and expression coefficients respectively. Next, by integrating audio features with a facial depth map, we provide comprehensive supervision for precise lip-audio synchronization in facial generation. Additionally, we constructed a Chinese talking-face dataset containing 130 hours of high-quality video. JoyGen is trained on the open-source HDTF dataset and our curated dataset. Experimental results demonstrate superior lip-audio synchronization and visual quality achieved by our method.
Paper Structure (15 sections, 3 equations, 5 figures, 2 tables)

This paper contains 15 sections, 3 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Dataset
  4. Method
  5. 3DMM
  6. Audio2Motion
  7. Depth map
  8. Edit talking face
  9. Training Details
  10. Experiments
  11. Data Preprocessing
  12. Experimental Setups
  13. Quantitative Results
  14. Qualitative Results
  15. Conclusions and Future Work

Figures (5)

  • Figure 1: Overview of the proposed method JoyGen. Top: Training pipeline, Bottom: Inference pipeline.
  • Figure 2: Statistics of our newly curated chinese talking-face dataset. Our dataset has an approximately equal ratio of males and females, with varying video lengths and frame rates, and includes high-resolution video frames.
  • Figure 3: The distribution curves of LSE-D and LSE-C scores on the HDTF dataset
  • Figure 4: The distribution curves of LSE-D and LSE-C scores on our collected dataset
  • Figure 5: Qualitative comparisons across different methods show that our approach significantly outperforms others in terms of lip synchronization and image quality.