Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hierarchical B-frame Video Coding for Long Group of Pictures

Ivan Kirillov, Denis Parkhomenko, Kirill Chernyshev, Alexander Pletnev, Yibo Shi, Kai Lin, Dmitry Babin

TL;DR

The paper tackles random-access video coding by proposing an end-to-end learned RA codec that trains on long frame sequences within a hierarchical GoP framework. It introduces two-reference B-frames, a Hierarchical Gain Unit for level-dependent latent scaling, and content-adaptation-based training to optimize rate-distortion across the hierarchy without transmitting adaptation parameters. Key contributions include training on GoP sizes up to 32, a random path sampling strategy for scalable training, and RA-specific loss design, achieving competitive YUV-PSNR BD-rates with VTM and superior VMAF BD-rates on JVET test sets. The approach demonstrates practical impact by narrowing the gap to traditional codecs in RA scenarios while offering advantages over existing end-to-end RA methods, with extensible future work in motion estimation, perceptual objectives, and data-driven optimization.

Abstract

Learned video compression methods already outperform VVC in the low-delay (LD) case, but the random-access (RA) scenario remains challenging. Most works on learned RA video compression either use HEVC as an anchor or compare it to VVC in specific test conditions, using RGB-PSNR metric instead of Y-PSNR and avoiding comprehensive evaluation. Here, we present an end-to-end learned video codec for random access that combines training on long sequences of frames, rate allocation designed for hierarchical coding and content adaptation on inference. We show that under common test conditions (JVET-CTC), it achieves results comparable to VTM (VVC reference software) in terms of YUV-PSNR BD-Rate on some classes of videos, and outperforms it on almost all test sets in terms of VMAF BD-Rate. On average it surpasses open LD and RA end-to-end solutions in terms of VMAF and YUV BD-Rates.

Hierarchical B-frame Video Coding for Long Group of Pictures

TL;DR

The paper tackles random-access video coding by proposing an end-to-end learned RA codec that trains on long frame sequences within a hierarchical GoP framework. It introduces two-reference B-frames, a Hierarchical Gain Unit for level-dependent latent scaling, and content-adaptation-based training to optimize rate-distortion across the hierarchy without transmitting adaptation parameters. Key contributions include training on GoP sizes up to 32, a random path sampling strategy for scalable training, and RA-specific loss design, achieving competitive YUV-PSNR BD-rates with VTM and superior VMAF BD-rates on JVET test sets. The approach demonstrates practical impact by narrowing the gap to traditional codecs in RA scenarios while offering advantages over existing end-to-end RA methods, with extensible future work in motion estimation, perceptual objectives, and data-driven optimization.

Abstract

Learned video compression methods already outperform VVC in the low-delay (LD) case, but the random-access (RA) scenario remains challenging. Most works on learned RA video compression either use HEVC as an anchor or compare it to VVC in specific test conditions, using RGB-PSNR metric instead of Y-PSNR and avoiding comprehensive evaluation. Here, we present an end-to-end learned video codec for random access that combines training on long sequences of frames, rate allocation designed for hierarchical coding and content adaptation on inference. We show that under common test conditions (JVET-CTC), it achieves results comparable to VTM (VVC reference software) in terms of YUV-PSNR BD-Rate on some classes of videos, and outperforms it on almost all test sets in terms of VMAF BD-Rate. On average it surpasses open LD and RA end-to-end solutions in terms of VMAF and YUV BD-Rates.
Paper Structure (25 sections, 5 equations, 9 figures, 7 tables)

This paper contains 25 sections, 5 equations, 9 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Learned Image Compression
  4. Learned Video Compression
  5. Method
  6. Overview
  7. Hierarchical Coding
  8. B-frame model
  9. Hierarchical Gain Unit
  10. Content Adaptation
  11. Training
  12. Experiments
  13. Training
  14. Testing in YUV Colorspace
  15. Testing in RGB Colorspace
  16. ...and 10 more sections

Figures (9)

  • Figure 1: Low-delay coding scheme when group of pictures (GoP) length equals 8. Arrows represent reference structure: each P-frame refers to the previous frame.
  • Figure 2: (a):GoP structure for the hierarchical coding scheme when the GoP size is 8. Arrows connect frames to their child frames (which use them as reference). Coding order may vary, but should always satisfy the condition that frame $x_t$ is coded before its child frames. (b):Random Path sampling strategy when the training GoP size is 8. Only frames with colored incoming/outgoing arrows are compressed.
  • Figure 3: Overview of the proposed B-frame compression model.
  • Figure 4: (a): B-frame Motion Predictor scheme. First, the motion flows $OFE(\widehat{x} _p , x_t)$ and $OFE(\widehat{x} _f , x_t)$ are predicted using the pretrained Optical Flow Estimator $OFE$. These flows are used as an initialization for the deformable alignment offsets within the Pixel-to-Feature Motion Predictor module. It generates motion features $M _p, M _f$ from the $F_t$, corresponding flows, and reference features. (b): Alignment module scheme. Deformable alignment offsets are initialized with the reconstructed motion features $\widehat{M} _p , \widehat{M} _f$ and applied to the corresponding reconstructed features $\widehat{F} _p , \widehat{F} _f$.
  • Figure 5: CFR module scheme. First, feature $\check F _t$ is calculated as $\check F _t = C_p \cdot \tilde{F} _p + C_f \cdot \tilde{F} _f + \widehat{R} _t$. Then it is refined using the Intra Aggregation block, producing $\widehat{F} _t$
  • ...and 4 more figures