A Coding Framework and Benchmark towards Low-Bitrate Video Understanding

TL;DR

This work tackles the challenge of reliable video understanding under low-bitrate compression by introducing Video Coding for Semantics (VCS), a hybrid framework that couples a traditional video codec with a data-emerged neural semantic stream learned in a self-supervised manner. A semantic stream encoder (Sem-Enc) and a latent fusion network (LFN) fuse semantic content with decoded pixels, guided by a bottleneck-based contrastive objective that uses a one-bit semantic representation. The approach yields substantial bitrate savings and improved performance across action recognition, multiple object tracking, and video object segmentation on eight datasets, and is supported by a comprehensive benchmark against traditional, learnable, and VCM methods. The results highlight practical benefits for machine-centric video analysis at low bitrates and establish a solid baseline for future semantic-video-coding research.

Abstract

Video compression is indispensable to most video analysis systems. Despite saving transportation bandwidth, it also deteriorates downstream video understanding tasks, especially at low-bitrate settings. To systematically investigate this problem, we first thoroughly review the previous methods, revealing that three principles, i.e., task-decoupled, label-free, and data-emerged semantic prior, are critical to a machine-friendly coding framework but are not fully satisfied so far. In this paper, we propose a traditional-neural mixed coding framework that simultaneously fulfills all these principles, by taking advantage of both traditional codecs and neural networks (NNs). On one hand, the traditional codecs can efficiently encode the pixel signal of videos but may distort the semantic information. On the other hand, highly non-linear NNs are proficient in condensing video semantics into a compact representation. The framework is optimized by ensuring that a transportation-efficient semantic representation of the video is preserved w.r.t. the coding procedure, which is spontaneously learned from unlabeled data in a self-supervised manner. The videos collaboratively decoded from two streams (codec and NN) are of rich semantics, as well as visually photo-realistic, empirically boosting several mainstream downstream video analysis task performances without any post-adaptation procedure. Furthermore, by introducing the attention mechanism and adaptive modeling scheme, the video semantic modeling ability of our approach is further enhanced. Finally, we build a low-bitrate video understanding benchmark with three downstream tasks on eight datasets, demonstrating the notable superiority of our approach. All codes, data, and models will be available at \url{https://github.com/tianyuan168326/VCS-Pytorch}.

Figures (17)

• Figure 1: Performance improvements on various tasks by applying the proposed VCS framework to different traditional video codecs. The bitcost of the proposed VCS framework is calculated by incorporating the additional semantic stream for a fair comparison.
• Figure 2: Overview of VCS framework. It includes a semantic stream$\mathcal{S}$ in addition to the video stream. $\mathcal{S}$ has no prior constraints (e.g., keypoint, edge, segmentation map) and is completely data-emerged. On the recipient side, the information within two streams is fused into the ultimately decoded video $\hat{X}$. $\hat{X}$ can be directly deployed to the downstream task models, which are not involved in the training procedure. The semantic completeness of $\hat{X}$ is ensured through a bottleneck map-based contrastive learning objective, wherein $\hat{X}$ is pulled closer to the original video, but pushed far away from the negative video. The bitcost of the semantic stream is also estimated and optimized to be minimal. Here, [$\cdot$] and $p$ represent the $round$ function and a learnable prior probability model, respectively. The H.265 codec is used as an example illustration.
• Figure 3: (a) Sem-Enc hierarchically transforms the input video $X$ and the difference map $D$ from pixel space to semantic space, producing the semantic feature $\mathcal{S}$. (b) DGFM is introduced to fuse the intermediate features from the two pathways of Sem-Enc. $\oplus$ and $\otimes$ denote the element-wise summation and multiplication, respectively. $\circledast$ denotes the pixel-adaptive convolution operation.
• Figure 4: Architecture of LFN. $\downarrow 4$ and $\uparrow 4$ denote the operations that downscale and upscale the spatial scale of the input by four times, respectively. An attention-based feature fusion module (F) is adopted for adaptively fusing the information in the down and up pathways. $\oplus$, $\otimes$ and © denote the element-wise summation, multiplication, and channel-wise concatenation, respectively.
• Figure 5: The contrastive learning procedure is applied to the decoded video $\hat{X}$, employing a one-bit bottleneck map (BM) representation. The original video $X$ serves as the positive sample, while $X^-$ represents a set of videos randomly sampled from the training dataset, utilized as negative samples for contrastive learning. For clarity, we illustrate with one negative video sample.