Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Video Compression with Hierarchical Temporal Neural Representation

Jun Zhu, Xinfeng Zhang, Lv Tang, Junhao Jiang, Gai Zhang, Jia Wang

TL;DR

TeNeRV tackles the limited temporal modeling of prior INR-based video compression by introducing a hierarchical temporal representation that fuses information across adjacent frames and adapts to GoP structure. The Inter-Frame Feature Fusion (IFF) captures local temporal coherence, while the GoP-Adaptive Modulation (GAM) learns group-specific priors and modulates network behavior across GoPs. A two-stage training scheme, adaptive GoP partitioning, and GoP-specific depthwise kernels balance expressiveness with parameter efficiency. Experiments on UVG and HEVC ClassB demonstrate superior rate-distortion performance and temporal stability compared to prior INR methods and traditional codecs, with ablations validating each component. This approach suggests a practical path to efficient neural video compression by explicitly modeling temporal hierarchies within implicit representations.

Abstract

Video compression has recently benefited from implicit neural representations (INRs), which model videos as continuous functions. INRs offer compact storage and flexible reconstruction, providing a promising alternative to traditional codecs. However, most existing INR-based methods treat the temporal dimension as an independent input, limiting their ability to capture complex temporal dependencies. To address this, we propose a Hierarchical Temporal Neural Representation for Videos, TeNeRV. TeNeRV integrates short- and long-term dependencies through two key components. First, an Inter-Frame Feature Fusion (IFF) module aggregates features from adjacent frames, enforcing local temporal coherence and capturing fine-grained motion. Second, a GoP-Adaptive Modulation (GAM) mechanism partitions videos into Groups-of-Pictures and learns group-specific priors. The mechanism modulates network parameters, enabling adaptive representations across different GoPs. Extensive experiments demonstrate that TeNeRV consistently outperforms existing INR-based methods in rate-distortion performance, validating the effectiveness of our proposed approach.

Video Compression with Hierarchical Temporal Neural Representation

TL;DR

TeNeRV tackles the limited temporal modeling of prior INR-based video compression by introducing a hierarchical temporal representation that fuses information across adjacent frames and adapts to GoP structure. The Inter-Frame Feature Fusion (IFF) captures local temporal coherence, while the GoP-Adaptive Modulation (GAM) learns group-specific priors and modulates network behavior across GoPs. A two-stage training scheme, adaptive GoP partitioning, and GoP-specific depthwise kernels balance expressiveness with parameter efficiency. Experiments on UVG and HEVC ClassB demonstrate superior rate-distortion performance and temporal stability compared to prior INR methods and traditional codecs, with ablations validating each component. This approach suggests a practical path to efficient neural video compression by explicitly modeling temporal hierarchies within implicit representations.

Abstract

Video compression has recently benefited from implicit neural representations (INRs), which model videos as continuous functions. INRs offer compact storage and flexible reconstruction, providing a promising alternative to traditional codecs. However, most existing INR-based methods treat the temporal dimension as an independent input, limiting their ability to capture complex temporal dependencies. To address this, we propose a Hierarchical Temporal Neural Representation for Videos, TeNeRV. TeNeRV integrates short- and long-term dependencies through two key components. First, an Inter-Frame Feature Fusion (IFF) module aggregates features from adjacent frames, enforcing local temporal coherence and capturing fine-grained motion. Second, a GoP-Adaptive Modulation (GAM) mechanism partitions videos into Groups-of-Pictures and learns group-specific priors. The mechanism modulates network parameters, enabling adaptive representations across different GoPs. Extensive experiments demonstrate that TeNeRV consistently outperforms existing INR-based methods in rate-distortion performance, validating the effectiveness of our proposed approach.
Paper Structure (17 sections, 5 equations, 4 figures, 3 tables, 1 algorithm)

This paper contains 17 sections, 5 equations, 4 figures, 3 tables, 1 algorithm.

Table of Contents

  1. introduction
  2. Related work
  3. Video Compression
  4. INR-based Video Compression
  5. Method
  6. Inter-frame Feature Fusion
  7. GoP-Adpative Modulation
  8. Training Strategy
  9. experiments
  10. Set Up
  11. Dataset
  12. Implementation Details
  13. Video Representation
  14. Time Complexity
  15. Video Compression
  16. ...and 2 more sections

Figures (4)

  • Figure 1: Reconstruction quality (PSNR) with different training times under 3M and 6M model size. TeNeRV(Ours) achieves the best performance.
  • Figure 2: Overview of proposed TeNeRV architecture. The temporal fused grid is combined with the GoP-level grid and then upsampled through N TeNeRV blocks. The weights of these block are adaptive across different GoPs.
  • Figure 3: Video Compression results on the HEVC ClassB dataset.
  • Figure 4: Visualization of video compression results on different videos. Reconstruction detail are shown in the rigion marked by red box. The motion area (blue box) is presented by stacking pixels at the same position across consecutive frames.