Efficient Video Face Enhancement with Enhanced Spatial-Temporal Consistency

TL;DR

Experiments demonstrate that the proposed novel and efficient blind video face enhancement method surpasses the current state-of-the-art blind face video restoration and de-flickering methods on both efficiency and effectiveness.

Abstract

As a very common type of video, face videos often appear in movies, talk shows, live broadcasts, and other scenes. Real-world online videos are often plagued by degradations such as blurring and quantization noise, due to the high compression ratio caused by high communication costs and limited transmission bandwidth. These degradations have a particularly serious impact on face videos because the human visual system is highly sensitive to facial details. Despite the significant advancement in video face enhancement, current methods still suffer from $i)$ long processing time and $ii)$ inconsistent spatial-temporal visual effects (e.g., flickering). This study proposes a novel and efficient blind video face enhancement method to overcome the above two challenges, restoring high-quality videos from their compressed low-quality versions with an effective de-flickering mechanism. In particular, the proposed method develops upon a 3D-VQGAN backbone associated with spatial-temporal codebooks recording high-quality portrait features and residual-based temporal information. We develop a two-stage learning framework for the model. In Stage \Rmnum{1}, we learn the model with a regularizer mitigating the codebook collapse problem. In Stage \Rmnum{2}, we learn two transformers to lookup code from the codebooks and further update the encoder of low-quality videos. Experiments conducted on the VFHQ-Test dataset demonstrate that our method surpasses the current state-of-the-art blind face video restoration and de-flickering methods on both efficiency and effectiveness. Code is available at \url{https://github.com/Dixin-Lab/BFVR-STC}.

Figures (13)

• Figure 1: Network architecture of Stage I. Stage I uses HQ face videos to train HQ 3D-VQGAN ($E_h$ and $D_h$) and spatial and temporal codebooks ($\mathcal{C}_S$ and $\mathcal{C}_T$). (a) illustrates the quantization operation STLookUp through two codebooks in our proposed framework. (b) and (c) display the computation process of temporal attention and motion residual, respectively. (d) We leverage a pre-trained feature network DINOv2 and trainable multi-scale discriminator heads to construct a more powerful discriminator for stable training.
• Figure 2: Network architecture of Stage ii@. Stage ii@ uses HQ-LQ face video pairs to train LQ encoder $E_l$ and LookUp Transformers ($\mathcal{T}_S$ and $\mathcal{T}_T$). The weights of $D_h$ are pre-trained in Stage i@ and fixed in Stage ii@.
• Figure 3: Qualitative comparison on the VFHQ-Test for BFVR task. Our method has better fidelity and fewer hallucination cases compared to other methods, such as wrinkles (1st row), eye orientation (1st row), nose shape (2nd row), and hairstyle (3rd row).
• Figure 4: Comparison of temporal profile for brightness and pixel de-flickering. We select a column to observe the changes across time.
• Figure 5: Comparison of HTML]B5739Dspatial and HTML]7EA6E0temporal codebooks' utilization when applying different regualizations.