Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bridging the gap between image coding for machines and humans

Nam Le, Honglei Zhang, Francesco Cricri, Ramin G. Youvalari, Hamed Rezazadegan Tavakoli, Emre Aksu, Miska M. Hannuksela, Esa Rahtu

TL;DR

ICM codecs compress images for machine analysis but often degrade human-viewed quality due to checkerboard artifacts at low bitrates. The authors introduce a PatchGAN-based finetuning scheme that updates only the decoder of a pretrained ICM codec, keeping the encoder fixed to avoid bitrate changes while guiding outputs toward artifact-free visuals. This approach yields significant improvements in visual quality (PSNR/SSIM) with only negligible or controllable impact on machine-task performance, and it avoids the need for additional bitstreams. The method provides a practical path to bridge machine-oriented compression with human viewing requirements, reducing artifacts without extra cost and with potential for fine-grained control over the trade-off between visuals and task accuracy.

Abstract

Image coding for machines (ICM) aims at reducing the bitrate required to represent an image while minimizing the drop in machine vision analysis accuracy. In many use cases, such as surveillance, it is also important that the visual quality is not drastically deteriorated by the compression process. Recent works on using neural network (NN) based ICM codecs have shown significant coding gains against traditional methods; however, the decompressed images, especially at low bitrates, often contain checkerboard artifacts. We propose an effective decoder finetuning scheme based on adversarial training to significantly enhance the visual quality of ICM codecs, while preserving the machine analysis accuracy, without adding extra bitcost or parameters at the inference phase. The results show complete removal of the checkerboard artifacts at the negligible cost of -1.6% relative change in task performance score. In the cases where some amount of artifacts is tolerable, such as when machine consumption is the primary target, this technique can enhance both pixel-fidelity and feature-fidelity scores without losing task performance.

Bridging the gap between image coding for machines and humans

TL;DR

ICM codecs compress images for machine analysis but often degrade human-viewed quality due to checkerboard artifacts at low bitrates. The authors introduce a PatchGAN-based finetuning scheme that updates only the decoder of a pretrained ICM codec, keeping the encoder fixed to avoid bitrate changes while guiding outputs toward artifact-free visuals. This approach yields significant improvements in visual quality (PSNR/SSIM) with only negligible or controllable impact on machine-task performance, and it avoids the need for additional bitstreams. The method provides a practical path to bridge machine-oriented compression with human viewing requirements, reducing artifacts without extra cost and with potential for fine-grained control over the trade-off between visuals and task accuracy.

Abstract

Image coding for machines (ICM) aims at reducing the bitrate required to represent an image while minimizing the drop in machine vision analysis accuracy. In many use cases, such as surveillance, it is also important that the visual quality is not drastically deteriorated by the compression process. Recent works on using neural network (NN) based ICM codecs have shown significant coding gains against traditional methods; however, the decompressed images, especially at low bitrates, often contain checkerboard artifacts. We propose an effective decoder finetuning scheme based on adversarial training to significantly enhance the visual quality of ICM codecs, while preserving the machine analysis accuracy, without adding extra bitcost or parameters at the inference phase. The results show complete removal of the checkerboard artifacts at the negligible cost of -1.6% relative change in task performance score. In the cases where some amount of artifacts is tolerable, such as when machine consumption is the primary target, this technique can enhance both pixel-fidelity and feature-fidelity scores without losing task performance.
Paper Structure (9 sections, 4 equations, 2 figures, 1 table)

This paper contains 9 sections, 4 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related work
  3. Proposed method
  4. Baseline image codec model
  5. Codec finetuning with PatchGAN
  6. Experiments
  7. Experimental setup
  8. Experimental results
  9. Conclusions

Figures (2)

  • Figure 1: The overview of the finetuning scheme using PatchGAN discriminator. The dashed lines denote gradient back-propagation flows and dotted boxes denote the parameters getting updated by the optimizer. The green lines indicate the data from the input of the ICM codec and the red lines indicate the data from the output of the ICM codec.
  • Figure 2: The codec finetuned with PatchGAN (middle) effectively removes the checkerboard artifacts commonly found in the decoded images of the NN-based convolutional codec such as the base model (left), while the codec finetuned with limited adversarial impact (right) only mildly suppresses the artifacts. More examples available at: https://flysofast.github.io/human-finetuned-icm/.