Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Data standardization for robust lip sync

Chun Wang

TL;DR

The paper tackles lip-sync robustness under real-world distractors by introducing a data standardization pipeline (DSP) that leverages a 3D Morphable Model to disentangle lip motion from confounding factors. By training an expressional encoder (E-Net) and rendering standardized expressive images with controlled attributes, DSP provides inputs that yield higher data efficiency for existing lip-sync methods and competitive active speaker detection on ASW2021. Key contributions include a novel training dataset construction strategy with cross-collection constraints, a multi-term loss to enforce accurate expression disentanglement, and a face-synthesis framework that can optionally produce depth information. The approach demonstrates that domain-knowledge-based standardization can significantly improve robustness and generalization in AV tasks beyond purely data-driven augmentation.

Abstract

Lip sync is a fundamental audio-visual task. However, existing lip sync methods fall short of being robust in the wild. One important cause could be distracting factors on the visual input side, making extracting lip motion information difficult. To address these issues, this paper proposes a data standardization pipeline to standardize the visual input for lip sync. Based on recent advances in 3D face reconstruction, we first create a model that can consistently disentangle lip motion information from the raw images. Then, standardized images are synthesized with disentangled lip motion information, with all other attributes related to distracting factors set to predefined values independent of the input, to reduce their effects. Using synthesized images, existing lip sync methods improve their data efficiency and robustness, and they achieve competitive performance for the active speaker detection task.

Data standardization for robust lip sync

TL;DR

The paper tackles lip-sync robustness under real-world distractors by introducing a data standardization pipeline (DSP) that leverages a 3D Morphable Model to disentangle lip motion from confounding factors. By training an expressional encoder (E-Net) and rendering standardized expressive images with controlled attributes, DSP provides inputs that yield higher data efficiency for existing lip-sync methods and competitive active speaker detection on ASW2021. Key contributions include a novel training dataset construction strategy with cross-collection constraints, a multi-term loss to enforce accurate expression disentanglement, and a face-synthesis framework that can optionally produce depth information. The approach demonstrates that domain-knowledge-based standardization can significantly improve robustness and generalization in AV tasks beyond purely data-driven augmentation.

Abstract

Lip sync is a fundamental audio-visual task. However, existing lip sync methods fall short of being robust in the wild. One important cause could be distracting factors on the visual input side, making extracting lip motion information difficult. To address these issues, this paper proposes a data standardization pipeline to standardize the visual input for lip sync. Based on recent advances in 3D face reconstruction, we first create a model that can consistently disentangle lip motion information from the raw images. Then, standardized images are synthesized with disentangled lip motion information, with all other attributes related to distracting factors set to predefined values independent of the input, to reduce their effects. Using synthesized images, existing lip sync methods improve their data efficiency and robustness, and they achieve competitive performance for the active speaker detection task.
Paper Structure (15 sections, 8 equations, 4 figures, 3 tables)

This paper contains 15 sections, 8 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. RELATED WORK
  3. PRELIMINARIES
  4. APPROACH
  5. Overview of DSP
  6. Training Dataset Construction
  7. Model Architecture
  8. Loss Functions
  9. Training Strategy
  10. Face Synthesis
  11. EXPERIMENTS
  12. Datasets
  13. Results on Expression Disentanglement
  14. Results on Audio-Visual Tasks
  15. CONCLUSION

Figures (4)

  • Figure 1: A domain knowledge-based model of the main factors that affect videos, with miscellaneous factors like occlusion omitted.
  • Figure 2: Lip sync with DSP schematic.
  • Figure 3: Overview of the proposed approach.
  • Figure 4: Snapshots of synthesized data. (a) raw images; (b)-(d) synthesized RGB, with MGCNet estimated id, Deep3D estimated id, and Deep3D estimated id and exp, respectively; (e)-(f) synthesized RGBD with E-Net estimated exp.