Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Uncertainty-Aware Deep Video Compression with Ensembles

Wufei Ma, Jiahao Li, Bin Li, Yan Lu

TL;DR

This work proposes an uncertainty-aware video compression model that can effectively capture the predictive uncertainty with deep ensembles and introduces an ensemble-aware loss to encourage the diversity among ensemble members and investigate the benefits of incorporating adversarial training in the video compression task.

Abstract

Deep learning-based video compression is a challenging task, and many previous state-of-the-art learning-based video codecs use optical flows to exploit the temporal correlation between successive frames and then compress the residual error. Although these two-stage models are end-to-end optimized, the epistemic uncertainty in the motion estimation and the aleatoric uncertainty from the quantization operation lead to errors in the intermediate representations and introduce artifacts in the reconstructed frames. This inherent flaw limits the potential for higher bit rate savings. To address this issue, we propose an uncertainty-aware video compression model that can effectively capture the predictive uncertainty with deep ensembles. Additionally, we introduce an ensemble-aware loss to encourage the diversity among ensemble members and investigate the benefits of incorporating adversarial training in the video compression task. Experimental results on 1080p sequences show that our model can effectively save bits by more than 20% compared to DVC Pro.

Uncertainty-Aware Deep Video Compression with Ensembles

TL;DR

This work proposes an uncertainty-aware video compression model that can effectively capture the predictive uncertainty with deep ensembles and introduces an ensemble-aware loss to encourage the diversity among ensemble members and investigate the benefits of incorporating adversarial training in the video compression task.

Abstract

Deep learning-based video compression is a challenging task, and many previous state-of-the-art learning-based video codecs use optical flows to exploit the temporal correlation between successive frames and then compress the residual error. Although these two-stage models are end-to-end optimized, the epistemic uncertainty in the motion estimation and the aleatoric uncertainty from the quantization operation lead to errors in the intermediate representations and introduce artifacts in the reconstructed frames. This inherent flaw limits the potential for higher bit rate savings. To address this issue, we propose an uncertainty-aware video compression model that can effectively capture the predictive uncertainty with deep ensembles. Additionally, we introduce an ensemble-aware loss to encourage the diversity among ensemble members and investigate the benefits of incorporating adversarial training in the video compression task. Experimental results on 1080p sequences show that our model can effectively save bits by more than 20% compared to DVC Pro.
Paper Structure (13 sections, 12 equations, 9 figures, 3 tables, 2 algorithms)

This paper contains 13 sections, 12 equations, 9 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Uncertainty-Aware Deep Video Compression
  4. Uncertainties in Deep Video Compression
  5. Ensemble-Based Decoder
  6. Ensemble-Aware Training
  7. Adversarial Training with FGSM
  8. Experiments
  9. Experimental Setup
  10. Quantitative Results
  11. Qualitative Results
  12. Ablation Study
  13. Discussion

Figures (9)

  • Figure 1: (a) A low latency predictive coding-based video compression framework. (b) We follow the predictive coding-based video compression and propose ensemble-based decoders.
  • Figure 2: A preliminary experiment on the underlying uncertainty of the optical flows. (a) The current frame $x_t$ to be compressed. (b) The estimated MV $f_t$. (c) The decoded MV $\hat{f}_t$. (d) Aleatoric uncertainty measured as the L2 distance between two optical flows with and without a small perturbation on the bitstream. (e) Epistemic uncertainty measured by motion vectors that cannot be estimated well. (f) The predictive uncertainty represented by the ensemble-based decoder.
  • Figure 3: Rate-distortion comparisons between our model and x264, x265, DVC lu2019dvc , DVC Pro 9072487, Hu_ECCV20 hu2020improving, LU_ECCV20 lu2020content, Agustsson_CVPR20 Agustsson_2020_CVPR, and NeRV hao2021nerv on different datasets. veryslow preset is used for both x264 and x265. Best viewed in color.
  • Figure 4: (a) Effectiveness of various proposed modules. (b) Ablation study on the number of members in ensemble-based decoders.
  • Figure 5: Visualization of the predictive uncertainty represented by our proposed ensemble-based decoder on the first two frames in the BasketballDrive, RaceHorses, and Kimono1 sequence. The detailed calculations are presented in Eq. \ref{['eq:uncertainty']}.
  • ...and 4 more figures