Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SFQA: A Comprehensive Perceptual Quality Assessment Dataset for Singing Face Generation

Zhilin Gao, Yunhao Li, Sijing Wu, Yucheng Zhu, Huiyu Duan, Guangtao Zhai

TL;DR

SFQA tackles the lack of singing-face quality benchmarks by creating a dedicated SFG quality-assessment dataset. It compiles 5,184 videos generated from 100 reference portraits and 36 music clips across 7 styles using 12 generation methods, accompanied by subjective MOS ratings. The analysis reveals substantial variability in visual quality and audio-visual synchronization across methods and highlights limitations of existing objective QA metrics for SFG. The resource provides a standardized benchmark and guidance for developing SFG-specific quality measures, with implications for robust digital-human generation in singing contexts.

Abstract

The Talking Face Generation task has enormous potential for various applications in digital humans and agents, etc. Singing, as a common facial movement second only to talking, can be regarded as a universal language across ethnicities and cultures. However, it is often underestimated in the field due to lack of singing face datasets and the domain gap between singing and talking in rhythm and amplitude. More significantly, the quality of Singing Face Generation (SFG) often falls short and is uneven or limited by different applicable scenarios, which prompts us to propose timely and effective quality assessment methods to ensure user experience. To address existing gaps in this domain, this paper introduces a new SFG content quality assessment dataset SFQA, built using 12 representative generation methods. During the construction of the dataset, 100 photographs or portraits, as well as 36 music clips from 7 different styles, are utilized to generate 5,184 singing face videos that constitute the SFQA dataset. To further explore the quality of SFG methods, subjective quality assessment is conducted by evaluators, whose ratings reveal a significant variation in quality among different generation methods. Based on our proposed SFQA dataset, we comprehensively benchmark the current objective quality assessment algorithms.

SFQA: A Comprehensive Perceptual Quality Assessment Dataset for Singing Face Generation

TL;DR

SFQA tackles the lack of singing-face quality benchmarks by creating a dedicated SFG quality-assessment dataset. It compiles 5,184 videos generated from 100 reference portraits and 36 music clips across 7 styles using 12 generation methods, accompanied by subjective MOS ratings. The analysis reveals substantial variability in visual quality and audio-visual synchronization across methods and highlights limitations of existing objective QA metrics for SFG. The resource provides a standardized benchmark and guidance for developing SFG-specific quality measures, with implications for robust digital-human generation in singing contexts.

Abstract

The Talking Face Generation task has enormous potential for various applications in digital humans and agents, etc. Singing, as a common facial movement second only to talking, can be regarded as a universal language across ethnicities and cultures. However, it is often underestimated in the field due to lack of singing face datasets and the domain gap between singing and talking in rhythm and amplitude. More significantly, the quality of Singing Face Generation (SFG) often falls short and is uneven or limited by different applicable scenarios, which prompts us to propose timely and effective quality assessment methods to ensure user experience. To address existing gaps in this domain, this paper introduces a new SFG content quality assessment dataset SFQA, built using 12 representative generation methods. During the construction of the dataset, 100 photographs or portraits, as well as 36 music clips from 7 different styles, are utilized to generate 5,184 singing face videos that constitute the SFQA dataset. To further explore the quality of SFG methods, subjective quality assessment is conducted by evaluators, whose ratings reveal a significant variation in quality among different generation methods. Based on our proposed SFQA dataset, we comprehensively benchmark the current objective quality assessment algorithms.
Paper Structure (13 sections, 4 figures, 2 tables)

This paper contains 13 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Talking Face Generation
  4. Quality Assessment Method
  5. Dataset Construction
  6. Material Collection
  7. Selected Generation Methods
  8. Subjective Experiment
  9. MOSs Analysis
  10. benchmark experiment
  11. Experimental Settings
  12. Performance & Analysis
  13. conclusion

Figures (4)

  • Figure 1: The construction process of SFQA dataset and diverse raw materials.
  • Figure 2: Typical quality issues in contemporary TFG (Left: original reference image example)
  • Figure 3: User-rating GUI for subjective evaluation.
  • Figure 4: The distribution of subjective scores: (Left) Visual Perception Quality • (Right) Audio-Visual Consistency