Embedding Compression Distortion in Video Coding for Machines

TL;DR

This work tackles the problem that traditional video codecs optimize for pixel-domain and human-perception metrics, often harming machine perception tasks. It introduces Compression Distortion Representation Embedding (CDRE), which extracts distortion in the feature domain, compresses it with a lightweight codec, and progressively embeds it into downstream models to inform inference about current compression distortions. The approach uses a compression-sensitive extractor, a VAE-like distortion encoder/decoder with linear modulation, binary quantization, and CNN/Transformer-aware embedding, optimized by a loss that combines task performance with feature-domain distortion emphasis. Empirical results across three downstream tasks and multiple codecs show that CDRE substantially improves rate-task performance with modest bitrate and computation overhead, and demonstrates strong transferability across codecs and VC M scenarios.

Abstract

Currently, video transmission serves not only the Human Visual System (HVS) for viewing but also machine perception for analysis. However, existing codecs are primarily optimized for pixel-domain and HVS-perception metrics rather than the needs of machine vision tasks. To address this issue, we propose a Compression Distortion Representation Embedding (CDRE) framework, which extracts machine-perception-related distortion representation and embeds it into downstream models, addressing the information lost during compression and improving task performance. Specifically, to better analyze the machine-perception-related distortion, we design a compression-sensitive extractor that identifies compression degradation in the feature domain. For efficient transmission, a lightweight distortion codec is introduced to compress the distortion information into a compact representation. Subsequently, the representation is progressively embedded into the downstream model, enabling it to be better informed about compression degradation and enhancing performance. Experiments across various codecs and downstream tasks demonstrate that our framework can effectively boost the rate-task performance of existing codecs with minimal overhead in terms of bitrate, execution time, and number of parameters. Our codes and supplementary materials are released in https://github.com/Ws-Syx/CDRE/.

Figures (8)

• Figure 1: (a) Overview of our proposed CDRE framework, which extracts, compresses, transmits, and embeds machine-perception-related distortion representations. It helps the downstream model be aware of specific distortions in the input data, improving task accuracy. (b) Visualization of compact and binary representation of machine-perception-related distortion. Detection accuracy increases after distortion representation embedding.
• Figure 2: The process of distortion representation extraction and compression at the encoder side. $C$ denotes channel-wise concatenation, $Q$ denotes binary quantization, and $\times|+$ denotes linear modulation operations. Both the compressed image and the original image are input into the compression-sensitive extractor, yielding multi-level features $F^o_{1,2,3}$ and $F^c_{1,2,3}$ to identify distortions in the feature domain. Subsequently, under the modulation of the aforementioned features, the distortion encoder extracts and compresses the distortions into a compact representation, which is then quantized.
• Figure 3: Visual example from MS-COCO-2017 dataset. White means a higher value in the feature map. Since the feature distance between original and compressed images is amplified at three distinct scales, distortion is more easily discernible in the feature domain than in the pixel domain.
• Figure 4: The process of distortion representation reconstruction and embedding at the decoder side. The representation is reconstructed by the distortion decoder. Then it is transferred and embedded into the backbone of the downstream model for better performance. The structure of the distortion decoder and distortion transformation module are detailed in supplementary materials.
• Figure 5: The rate-task performance of (a) object detection and (b) Video instance segmentation across various existing HVS-oriented codecs. "Ours-FD" represents CDRE modules are optimized but the downstream model is frozen, "Ours" represents CDRE modules and the downstream model are jointly optimized. The average precision on uncompressed data is 37.3% (object detection) and 51.5% (instance segmentation).