Learned Scalable Video Coding For Humans and Machines

TL;DR

An end-to-end learnable video codec that supports a machine vision task in its base layer, while its enhancement layer, together with the base layer, supports input reconstruction for human viewing is introduced.

Abstract

Video coding has traditionally been developed to support services such as video streaming, videoconferencing, digital TV, and so on. The main intent was to enable human viewing of the encoded content. However, with the advances in deep neural networks (DNNs), encoded video is increasingly being used for automatic video analytics performed by machines. In applications such as automatic traffic monitoring, analytics such as vehicle detection, tracking and counting, would run continuously, while human viewing could be required occasionally to review potential incidents. To support such applications, a new paradigm for video coding is needed that will facilitate efficient representation and compression of video for both machine and human use in a scalable manner. In this manuscript, we introduce an end-to-end learnable video codec that supports a machine vision task in its base layer, while its enhancement layer, together with the base layer, supports input reconstruction for human viewing. The proposed system is constructed based on the concept of conditional coding to achieve better compression gains. Comprehensive experimental evaluations conducted on four standard video datasets demonstrate that our framework outperforms both state-of-the-art learned and conventional video codecs in its base layer, while maintaining comparable performance on the human vision task in its enhancement layer. Implementation of the proposed system is available at https://github.com/hadipardis/svc

Figures (21)

• Figure 1: Top: residual coding. Dashed lines are not used in residual coding, they are shown to aid the analysis in the text. Bottom: conditional coding.
• Figure 2: The block diagram of the proposed learned scalable video compression system. The input video frame is $x_t$. The base layer has two outputs: the base frame $\hat{x}_t^b$ and the latent features $\hat{r}_t$ by which the computer vision task is performed. The enhancement layer has only one output, enhancement frame, which is the reconstructed frame $\hat{x}_t$.
• Figure 3: The block diagram of the base layer of the proposed system. The base layer has two outputs: the reconstructed base frame $\hat{x}_t^b$ and the latent features $\hat{r}_t$ by which the computer vision task is performed. Our base layer's architecture is the same as our previous work LCCM-VC lccm. The Mode Generator is explained in more details in Fig. \ref{['fig:mode_generator']}.
• Figure 4: The structure of the mode generator.
• Figure 5: A visual example of a portion of the base frame at 0.11 bpp. The original frame $x_t$ (left) and the base frame $\hat{x}_t^b$ (right).