Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond GFVC: A Progressive Face Video Compression Framework with Adaptive Visual Tokens

Bolin Chen, Shanzhi Yin, Zihan Zhang, Jie Chen, Ru-Ling Liao, Lingyu Zhu, Shiqi Wang, Yan Ye

TL;DR

Experimental results demonstrate that the proposed PFVC framework can achieve better coding flexibility and superior rate-distortion performance in comparison with the latest Versatile Video Coding (VVC) codec and the state-of-the-art GFVC algorithms.

Abstract

Recently, deep generative models have greatly advanced the progress of face video coding towards promising rate-distortion performance and diverse application functionalities. Beyond traditional hybrid video coding paradigms, Generative Face Video Compression (GFVC) relying on the strong capabilities of deep generative models and the philosophy of early Model-Based Coding (MBC) can facilitate the compact representation and realistic reconstruction of visual face signal, thus achieving ultra-low bitrate face video communication. However, these GFVC algorithms are sometimes faced with unstable reconstruction quality and limited bitrate ranges. To address these problems, this paper proposes a novel Progressive Face Video Compression framework, namely PFVC, that utilizes adaptive visual tokens to realize exceptional trade-offs between reconstruction robustness and bandwidth intelligence. In particular, the encoder of the proposed PFVC projects the high-dimensional face signal into adaptive visual tokens in a progressive manner, whilst the decoder can further reconstruct these adaptive visual tokens for motion estimation and signal synthesis with different granularity levels. Experimental results demonstrate that the proposed PFVC framework can achieve better coding flexibility and superior rate-distortion performance in comparison with the latest Versatile Video Coding (VVC) codec and the state-of-the-art GFVC algorithms. The project page can be found at https://github.com/Berlin0610/PFVC.

Beyond GFVC: A Progressive Face Video Compression Framework with Adaptive Visual Tokens

TL;DR

Experimental results demonstrate that the proposed PFVC framework can achieve better coding flexibility and superior rate-distortion performance in comparison with the latest Versatile Video Coding (VVC) codec and the state-of-the-art GFVC algorithms.

Abstract

Recently, deep generative models have greatly advanced the progress of face video coding towards promising rate-distortion performance and diverse application functionalities. Beyond traditional hybrid video coding paradigms, Generative Face Video Compression (GFVC) relying on the strong capabilities of deep generative models and the philosophy of early Model-Based Coding (MBC) can facilitate the compact representation and realistic reconstruction of visual face signal, thus achieving ultra-low bitrate face video communication. However, these GFVC algorithms are sometimes faced with unstable reconstruction quality and limited bitrate ranges. To address these problems, this paper proposes a novel Progressive Face Video Compression framework, namely PFVC, that utilizes adaptive visual tokens to realize exceptional trade-offs between reconstruction robustness and bandwidth intelligence. In particular, the encoder of the proposed PFVC projects the high-dimensional face signal into adaptive visual tokens in a progressive manner, whilst the decoder can further reconstruct these adaptive visual tokens for motion estimation and signal synthesis with different granularity levels. Experimental results demonstrate that the proposed PFVC framework can achieve better coding flexibility and superior rate-distortion performance in comparison with the latest Versatile Video Coding (VVC) codec and the state-of-the-art GFVC algorithms. The project page can be found at https://github.com/Berlin0610/PFVC.

Paper Structure

This paper contains 9 sections, 8 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. The Proposed PFVC Framework
  3. Overall Encoding/Decoding Processes
  4. Key Techniques Introduction
  5. Progressive Model Training Strategy
  6. Experimental Results
  7. Experimental Settings
  8. Performance Comparisons
  9. Conclusion

Figures (4)

  • Figure 1: Overview of the proposed PFVC framework towards bandwidth intelligence.
  • Figure 2: Overall RD performance comparisons with VVC bross2021overview, FOMM FOMM and CFTE CHEN2022DCC in terms of DISTS, LPIPS, FVD, MANIQA, PSNR and SSIM.
  • Figure 3: Subjective comparisons of reconstruction quality using the proposed PFVC with 4 different token granularities. More visual examples can be found in https://github.com/Berlin0610/PFVC.
  • Figure 4: Subjective comparisons of reconstruction quality using different compression algorithms like VVC bross2021overview, FOMM FOMM and CFTE CHEN2022DCC and our proposed PFVC at 2 different bitrate levels. More visual examples can be found in https://github.com/Berlin0610/PFVC.