Symmetric Entropy-Constrained Video Coding for Machines

TL;DR

The paper tackles video coding for machines by proposing SEC-VCM, a framework that explicitly aligns the video codec with a visual backbone (VB) or visual foundation model (VFM) through a symmetric entropy-constrained mechanism (BiEC) and a semantic-pixel fusion (SPDF) module. The approach recovers machine-oriented semantic features via an information-alignment-based semantic decoder and enforces entropy-based symmetry between VB and the decoder, using a two-stage training regime with losses that couple rate, distortion, and cross-model consistency: $L_{ ext{total}} = L_{ ext{base}} + oldsymbol{l}L_e + oldsymbol{l}_{ ext{cons}}L_{ ext{cons}}$, where $L_e = H(f_{ ext{VB}}|f_{ ext{MVS}}) + H(f_{ ext{MVS}}|f_{ ext{VB}})$ and the conditional entropies guide information alignment. SPDF injects pixel-level priors to refine machine-oriented reconstruction, balancing semantic fidelity with perceptual quality. Experiments across object detection, video instance segmentation, video object segmentation, and multiple object tracking show substantial rate-task gains over VTM and competitive results versus contemporary neural codecs, demonstrating strong generalization and practical impact for multi-task machine vision workloads.

Abstract

As video transmission increasingly serves machine vision systems (MVS) instead of human vision systems (HVS), video coding for machines (VCM) has become a critical research topic. Existing VCM methods often bind codecs to specific downstream models, requiring retraining or supervised data, thus limiting generalization in multi-task scenarios. Recently, unified VCM frameworks have employed visual backbones (VB) and visual foundation models (VFM) to support multiple video understanding tasks with a single codec. They mainly utilize VB/VFM to maintain semantic consistency or suppress non-semantic information, but seldom explore how to directly link video coding with understanding under VB/VFM guidance. Hence, we propose a Symmetric Entropy-Constrained Video Coding framework for Machines (SEC-VCM). It establishes a symmetric alignment between the video codec and VB, allowing the codec to leverage VB's representation capabilities to preserve semantics and discard MVS-irrelevant information. Specifically, a bi-directional entropy-constraint (BiEC) mechanism ensures symmetry between the process of video decoding and VB encoding by suppressing conditional entropy. This helps the codec to explicitly handle semantic information beneficial to MVS while squeezing useless information. Furthermore, a semantic-pixel dual-path fusion (SPDF) module injects pixel-level priors into the final reconstruction. Through semantic-pixel fusion, it suppresses artifacts harmful to MVS and improves machine-oriented reconstruction quality. Experimental results show our framework achieves state-of-the-art~(SOTA) in rate-task performance, with significant bitrate savings over VTM on video instance segmentation (37.4%), video object segmentation (29.8%), object detection (46.2%), and multiple object tracking (44.9%). We will release our code soon.

Figures (13)

• Figure 1: Overview of different methods. $I(\cdot)$ denotes mutual information, $H(\cdot)$ denotes Shannon entropy, and "$\Uparrow$" denotes that the optimization goal is to maximize the value. $\mathcal{L}_a$ and $\mathcal{L}_b$ denote downstream task losses, $\mathcal{L}_{\textit{cons}}$ denotes loss for maintaining feature consistency, and $\mathcal{E}_{\textit{visual}}$ denotes the encoder of visual backbone (VB) or visual foundation model (VFM). (a) Standard or neural video codecs h264h265 are optimized for HVS, which makes the decoded videos $\hat{x}$ suboptimal for machine vision. (b) Each neural video/feature codec sfmasheng2023lvvcmisra2022video is optimized for a specific machine vision task by downstream task loss. (c) The unified codec smcsmc++sa_icmfree_vsc is optimized with the assistance of the VB or VFM to reconstruct videos suitable for multiple downstream tasks. (d) Our unified compression framework aligns the video coding process with the video understanding process from the entropy perspective and squeezes machine-vision-irrelevant information during the process of video reconstruction.
• Figure 2: Overview of the proposed SEC-VCM framework. (1) The basic compression loop contains motion and contextual coding processes. "Q" denotes the quantization operation. Notably, some modules in this step are not shown above for simplicity, since this step is directly adopted from the neural video codec DCVC-HEM dcvc_hem. (2) The semantic decoding process is information-aligned with the encoding process of the pretrained visual backbone by the proposed BiEC mechanism from an entropy perspective. This helps the semantic decoder $\mathcal{D}_{\textit{sem}}$ purely handle information beneficial to MVS. (3) The pixel-level prior knowledge $f_{\textit{pixel}}$ is injected into the perception-oriented decoding process to fulfill frame content, suppress artifacts, and enhance final reconstruction quality.
• Figure 3: (a) Structure of the proposed information-alignment-based semantic decoder and BiEC mechanism. $DE$ denotes the distribution estimation module, and "$\uparrow\times 2$" denotes two times up-sampling. During training, the intermediate features for the decoding $f_{\textit{MVS}}$ are made hierarchically symmetric with the features $f_{\textit{VB}}$ from the visual backbone. (b) Structure of the proposed SPDF module. Machine vision system (MVS)-oriented semantic feature $f_t^{\textit{MVS}}$ and pixel-related feature $f_t^{\textit{pixel}}$ are selectively combined based on the factor $\alpha$.
• Figure 4: The process of the BiEC mechanism. Information alignment between $f_{\textit{MVS}}$ for decoding and $f_{\textit{VB}}$ from the visual backbone is achieved through the constraint of conditional entropy in two distinct directions. "$\Downarrow$" indicates that the optimization goal is to minimize the item.
• Figure 5: The t-SNE visualization of intermediate features $f_{\textit{MVS}}^{1/8}$ in the semantic decoder without and with symmetric information alignment. After alignment via the BiEC mechanism, intra-class distances are effectively reduced, and the intermediate features $f_{\textit{MVS}}^{1/8}$ in the decoding process exhibit more discriminative semantic representations.