Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SingingHead: A Large-scale 4D Dataset for Singing Head Animation

Sijing Wu, Yunhao Li, Weitian Zhang, Jun Jia, Yucheng Zhu, Yichao Yan, Guangtao Zhai, Xiaokang Yang

TL;DR

This work tackles the gap in singing head animation by introducing SingingHead, a large-scale dataset with over 27 hours of synchronized singing video, 3D facial motion, singing audio, and background music from 76 subjects across 8 music styles. It also presents UniSinger, a unified framework that combines a CVAE-transformer based 3D facial animation module with a depth-to-portrait face renderer to deliver both 3D singing head animation and 2D singing portrait video synthesis. The authors benchmark existing 3D and 2D audio-driven methods on the singing task and demonstrate that training on SingingHead substantially improves performance, while UniSinger achieves competitive results on both 3D and 2D benchmarks and runs in real time. The dataset and framework advance the field by enabling high-fidelity, singing-specific facial animation and by providing benchmarks and a baseline for future research and commercial applications in virtual avatars and entertainment.

Abstract

Singing, as a common facial movement second only to talking, can be regarded as a universal language across ethnicities and cultures, plays an important role in emotional communication, art, and entertainment. However, it is often overlooked in the field of audio-driven facial animation due to the lack of singing head datasets and the domain gap between singing and talking in rhythm and amplitude. To this end, we collect a high-quality large-scale singing head dataset, SingingHead, which consists of more than 27 hours of synchronized singing video, 3D facial motion, singing audio, and background music from 76 individuals and 8 types of music. Along with the SingingHead dataset, we benchmark existing audio-driven 3D facial animation methods and 2D talking head methods on the singing task. Furthermore, we argue that 3D and 2D facial animation tasks can be solved together, and propose a unified singing head animation framework named UniSinger to achieve both singing audio-driven 3D singing head animation and 2D singing portrait video synthesis, which achieves competitive results on both 3D and 2D benchmarks. Extensive experiments demonstrate the significance of the proposed singing-specific dataset in promoting the development of singing head animation tasks, as well as the promising performance of our unified facial animation framework.

SingingHead: A Large-scale 4D Dataset for Singing Head Animation

TL;DR

This work tackles the gap in singing head animation by introducing SingingHead, a large-scale dataset with over 27 hours of synchronized singing video, 3D facial motion, singing audio, and background music from 76 subjects across 8 music styles. It also presents UniSinger, a unified framework that combines a CVAE-transformer based 3D facial animation module with a depth-to-portrait face renderer to deliver both 3D singing head animation and 2D singing portrait video synthesis. The authors benchmark existing 3D and 2D audio-driven methods on the singing task and demonstrate that training on SingingHead substantially improves performance, while UniSinger achieves competitive results on both 3D and 2D benchmarks and runs in real time. The dataset and framework advance the field by enabling high-fidelity, singing-specific facial animation and by providing benchmarks and a baseline for future research and commercial applications in virtual avatars and entertainment.

Abstract

Singing, as a common facial movement second only to talking, can be regarded as a universal language across ethnicities and cultures, plays an important role in emotional communication, art, and entertainment. However, it is often overlooked in the field of audio-driven facial animation due to the lack of singing head datasets and the domain gap between singing and talking in rhythm and amplitude. To this end, we collect a high-quality large-scale singing head dataset, SingingHead, which consists of more than 27 hours of synchronized singing video, 3D facial motion, singing audio, and background music from 76 individuals and 8 types of music. Along with the SingingHead dataset, we benchmark existing audio-driven 3D facial animation methods and 2D talking head methods on the singing task. Furthermore, we argue that 3D and 2D facial animation tasks can be solved together, and propose a unified singing head animation framework named UniSinger to achieve both singing audio-driven 3D singing head animation and 2D singing portrait video synthesis, which achieves competitive results on both 3D and 2D benchmarks. Extensive experiments demonstrate the significance of the proposed singing-specific dataset in promoting the development of singing head animation tasks, as well as the promising performance of our unified facial animation framework.
Paper Structure (17 sections, 4 equations, 11 figures, 4 tables)

This paper contains 17 sections, 4 equations, 11 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Audio-driven 3D Facial Animation
  4. Audio-driven 2D Portrait Animation
  5. 2D and 3D Facial Animation Datasets
  6. SingingHead Dataset
  7. Data Acquisition
  8. Data Processing
  9. Dataset Analysis
  10. Method
  11. VAE-based 3D Facial Animation
  12. Face Renderer
  13. Experiments
  14. Implementation details
  15. 3D Singing Head Animation
  16. ...and 2 more sections

Figures (11)

  • Figure 1: We present a new facial animation dataset, SingingHead, which contains more than 27 hours of synchronized singing video, 3D facial motion, singing audio, and background music (BGM) collected from 76 subjects. Along with the SingingHead dataset, we propose a unified framework, UniSinger, to generate both 3D facial motion and 2D singing portrait video according to the input singing audio.
  • Figure 2: Comparison between talking and singing. (a) and (b) are the bar charts of the dominant frequency distribution and BPM distribution of audio, respectively. Our singing dataset has a wider range of dominant frequency and BPM distribution compared to talking datasets. (c) shows the chromagrams of a talking audio and a singing audio, which are very different and show that singing has more regular and stable pitch patterns as well as organized energy distribution. (d) is the distribution of mouth opening size calculated from facial landmarks, which shows that the mouth size is typically larger when singing compared to talking. All these demonstrate that singing is quite different from talking, and therefore needs to be studied separately.
  • Figure 3: Data collection pipeline. We use microphone, laptop, Azure Kinect camera, and light stage to collect raw data including singing audio, original songs downloaded from the Internet, singing video, and 3D head scan. Then, we process the raw data to obtain the cleaned singing audio, background music, portrait video, and 3D facial motion, which make up our SingingHead dataset.
  • Figure 4: Data Statistics. We show the pie charts of (a) the music category distribution, (b) the professional distribution of volunteers, (c) the gender distribution of volunteers, and (d) the language distribution, respectively.
  • Figure 5: Visualization of SingingHead dataset. We show some key frames from singing portrait videos and the corresponding 3D facial motions.
  • ...and 6 more figures