Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accelerating Learnt Video Codecs with Gradient Decay and Layer-wise Distillation

Tianhao Peng, Ge Gao, Heming Sun, Fan Zhang, David Bull

TL;DR

This paper tackles the high decoding complexity of end-to-end learnt video codecs by introducing a model-agnostic pruning framework that combines gradient decay with adaptive layer-wise distillation to enforce global sparsity across decoder modules. Gradient decay gradually attenuates surrogate gradients for pruned weights, enabling early exploration and later stabilization, while adaptive layer-wise distillation guides the sparse student model through stage-wise feature distillation from a pre-trained teacher. The approach is validated on three popular learnt codecs—FVC, DCVC, and DCVC-HEM—achieving up to 65% reductions in MACs and about a 2x speed-up with less than 0.3 dB BD-PSNR loss, demonstrating strong practical impact for real-time decoding. The results underline the potential for practical deployment of learned video codecs on resource-constrained devices and point to future work on more granular sparsity patterns and alternative distillation metrics to further close the rate-distortion-complexity gap.

Abstract

In recent years, end-to-end learnt video codecs have demonstrated their potential to compete with conventional coding algorithms in term of compression efficiency. However, most learning-based video compression models are associated with high computational complexity and latency, in particular at the decoder side, which limits their deployment in practical applications. In this paper, we present a novel model-agnostic pruning scheme based on gradient decay and adaptive layer-wise distillation. Gradient decay enhances parameter exploration during sparsification whilst preventing runaway sparsity and is superior to the standard Straight-Through Estimation. The adaptive layer-wise distillation regulates the sparse training in various stages based on the distortion of intermediate features. This stage-wise design efficiently updates parameters with minimal computational overhead. The proposed approach has been applied to three popular end-to-end learnt video codecs, FVC, DCVC, and DCVC-HEM. Results confirm that our method yields up to 65% reduction in MACs and 2x speed-up with less than 0.3dB drop in BD-PSNR. Supporting code and supplementary material can be downloaded from: https://jasminepp.github.io/lightweightdvc/

Accelerating Learnt Video Codecs with Gradient Decay and Layer-wise Distillation

TL;DR

This paper tackles the high decoding complexity of end-to-end learnt video codecs by introducing a model-agnostic pruning framework that combines gradient decay with adaptive layer-wise distillation to enforce global sparsity across decoder modules. Gradient decay gradually attenuates surrogate gradients for pruned weights, enabling early exploration and later stabilization, while adaptive layer-wise distillation guides the sparse student model through stage-wise feature distillation from a pre-trained teacher. The approach is validated on three popular learnt codecs—FVC, DCVC, and DCVC-HEM—achieving up to 65% reductions in MACs and about a 2x speed-up with less than 0.3 dB BD-PSNR loss, demonstrating strong practical impact for real-time decoding. The results underline the potential for practical deployment of learned video codecs on resource-constrained devices and point to future work on more granular sparsity patterns and alternative distillation metrics to further close the rate-distortion-complexity gap.

Abstract

In recent years, end-to-end learnt video codecs have demonstrated their potential to compete with conventional coding algorithms in term of compression efficiency. However, most learning-based video compression models are associated with high computational complexity and latency, in particular at the decoder side, which limits their deployment in practical applications. In this paper, we present a novel model-agnostic pruning scheme based on gradient decay and adaptive layer-wise distillation. Gradient decay enhances parameter exploration during sparsification whilst preventing runaway sparsity and is superior to the standard Straight-Through Estimation. The adaptive layer-wise distillation regulates the sparse training in various stages based on the distortion of intermediate features. This stage-wise design efficiently updates parameters with minimal computational overhead. The proposed approach has been applied to three popular end-to-end learnt video codecs, FVC, DCVC, and DCVC-HEM. Results confirm that our method yields up to 65% reduction in MACs and 2x speed-up with less than 0.3dB drop in BD-PSNR. Supporting code and supplementary material can be downloaded from: https://jasminepp.github.io/lightweightdvc/
Paper Structure (7 sections, 7 equations, 3 figures, 2 tables)

This paper contains 7 sections, 7 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Overview
  4. Gradient Decay
  5. Iterative Pruning with Adaptive Distillation
  6. Experiments
  7. Conclusion

Figures (3)

  • Figure 1: Comparing the performance-complexity trade-offs on UVG for different learnt video codecs. Models pruned using our method are positioned on the upper-left side of the frontier estimated from the original models.
  • Figure 2: Left: Illustration of the iterative pruning process with layer-wise distillation. Only the modules on the decoder side, denoted in beige colour, are selected for structured pruning. In each stage, the network is pruned alongside the distillation of a subset of modules. Right: Structured pruning with normalised feature distillation.
  • Figure 3: A-B: RD performance, measured by PSNR, of compact models against the original dense models on UVG and MCL-JCV, respectively; C: impact of gradient decay and layer-wise distillation on improving sparse training.