Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Real-world Video Face Restoration: A New Benchmark

Ziyan Chen, Jingwen He, Xinqi Lin, Yu Qiao, Chao Dong

TL;DR

This work introduced new real-world datasets named FOS with a taxonomy of "Full, Occluded, and Side" faces from mainly video frames to study the applicability of current methods on videos, and benchmarked both the state-of-the-art BFR methods and the video super resolution (VSR) methods to comprehensively study current approaches.

Abstract

Blind face restoration (BFR) on images has significantly progressed over the last several years, while real-world video face restoration (VFR), which is more challenging for more complex face motions such as moving gaze directions and facial orientations involved, remains unsolved. Typical BFR methods are evaluated on privately synthesized datasets or self-collected real-world low-quality face images, which are limited in their coverage of real-world video frames. In this work, we introduced new real-world datasets named FOS with a taxonomy of "Full, Occluded, and Side" faces from mainly video frames to study the applicability of current methods on videos. Compared with existing test datasets, FOS datasets cover more diverse degradations and involve face samples from more complex scenarios, which helps to revisit current face restoration approaches more comprehensively. Given the established datasets, we benchmarked both the state-of-the-art BFR methods and the video super resolution (VSR) methods to comprehensively study current approaches, identifying their potential and limitations in VFR tasks. In addition, we studied the effectiveness of the commonly used image quality assessment (IQA) metrics and face IQA (FIQA) metrics by leveraging a subjective user study. With extensive experimental results and detailed analysis provided, we gained insights from the successes and failures of both current BFR and VSR methods. These results also pose challenges to current face restoration approaches, which we hope stimulate future advances in VFR research.

Towards Real-world Video Face Restoration: A New Benchmark

TL;DR

This work introduced new real-world datasets named FOS with a taxonomy of "Full, Occluded, and Side" faces from mainly video frames to study the applicability of current methods on videos, and benchmarked both the state-of-the-art BFR methods and the video super resolution (VSR) methods to comprehensively study current approaches.

Abstract

Blind face restoration (BFR) on images has significantly progressed over the last several years, while real-world video face restoration (VFR), which is more challenging for more complex face motions such as moving gaze directions and facial orientations involved, remains unsolved. Typical BFR methods are evaluated on privately synthesized datasets or self-collected real-world low-quality face images, which are limited in their coverage of real-world video frames. In this work, we introduced new real-world datasets named FOS with a taxonomy of "Full, Occluded, and Side" faces from mainly video frames to study the applicability of current methods on videos. Compared with existing test datasets, FOS datasets cover more diverse degradations and involve face samples from more complex scenarios, which helps to revisit current face restoration approaches more comprehensively. Given the established datasets, we benchmarked both the state-of-the-art BFR methods and the video super resolution (VSR) methods to comprehensively study current approaches, identifying their potential and limitations in VFR tasks. In addition, we studied the effectiveness of the commonly used image quality assessment (IQA) metrics and face IQA (FIQA) metrics by leveraging a subjective user study. With extensive experimental results and detailed analysis provided, we gained insights from the successes and failures of both current BFR and VSR methods. These results also pose challenges to current face restoration approaches, which we hope stimulate future advances in VFR research.
Paper Structure (11 sections, 1 equation, 6 figures, 6 tables)

This paper contains 11 sections, 1 equation, 6 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Benchmark Settings
  4. Dataset
  5. Evaluation
  6. Experiments
  7. Investigation into The Evaluation Metrics
  8. Benchmark Results on Image Datasets.
  9. Benchmark Results on Video Datasets.
  10. Subjective Comparison
  11. Conclusion

Figures (6)

  • Figure 1: The face restoration results achieved by CodeFormer codeformer on widely used real-world datasets (a) gfpgancodeformer and our proposed FOS datasets (b). (Zoom in for details)
  • Figure 2: Overview of the proposed FOS datasets.
  • Figure 3: Visual examples of different scores on our five-point grading system. The number of stars lit corresponds to the score rated. (Zoom in for details)
  • Figure 4: SROCC v.s PLCC results of 10 IQA/FIQA algorithms and proposed stability evaluation metric VIDD on FOS-real(#158) regarding realness and FOS-V(#108) regarding reconstruction performance and stability.
  • Figure 5: Qualitative comparison of both state-of-the-art BFR methods and VSR methods on FOS-real. (Zoom in for details)
  • ...and 1 more figures