Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Image Coding for Machines with Edge Information Learning Using Segment Anything

Takahiro Shindo, Kein Yamada, Taiju Watanabe, Hiroshi Watanabe

TL;DR

SA-ICM introduces edge-focused image coding by training a learned image compression model to encode edge information derived from SAM, enabling strong machine-recognition performance while enhancing privacy through the removal of facial textures. The approach extends to video with SA-NeRV, which embeds edge structures into neural video representations to improve recognition accuracy. Empirical results show SA-ICM outperforms traditional RL-based ICM in object detection and segmentation while preserving background content, and SA-NeRV surpasses ordinary NeRV in video-based recognition tasks. This edge-information learning framework provides a privacy-preserving, robust solution for image and video compression tailored to machines, with code available publicly.

Abstract

Image Coding for Machines (ICM) is an image compression technique for image recognition. This technique is essential due to the growing demand for image recognition AI. In this paper, we propose a method for ICM that focuses on encoding and decoding only the edge information of object parts in an image, which we call SA-ICM. This is an Learned Image Compression (LIC) model trained using edge information created by Segment Anything. Our method can be used for image recognition models with various tasks. SA-ICM is also robust to changes in input data, making it effective for a variety of use cases. Additionally, our method provides benefits from a privacy point of view, as it removes human facial information on the encoder's side, thus protecting one's privacy. Furthermore, this LIC model training method can be used to train Neural Representations for Videos (NeRV), which is a video compression model. By training NeRV using edge information created by Segment Anything, it is possible to create a NeRV that is effective for image recognition (SA-NeRV). Experimental results confirm the advantages of SA-ICM, presenting the best performance in image compression for image recognition. We also show that SA-NeRV is superior to ordinary NeRV in video compression for machines. Code is available at https://github.com/final-0/SA-ICM.

Image Coding for Machines with Edge Information Learning Using Segment Anything

TL;DR

SA-ICM introduces edge-focused image coding by training a learned image compression model to encode edge information derived from SAM, enabling strong machine-recognition performance while enhancing privacy through the removal of facial textures. The approach extends to video with SA-NeRV, which embeds edge structures into neural video representations to improve recognition accuracy. Empirical results show SA-ICM outperforms traditional RL-based ICM in object detection and segmentation while preserving background content, and SA-NeRV surpasses ordinary NeRV in video-based recognition tasks. This edge-information learning framework provides a privacy-preserving, robust solution for image and video compression tailored to machines, with code available publicly.

Abstract

Image Coding for Machines (ICM) is an image compression technique for image recognition. This technique is essential due to the growing demand for image recognition AI. In this paper, we propose a method for ICM that focuses on encoding and decoding only the edge information of object parts in an image, which we call SA-ICM. This is an Learned Image Compression (LIC) model trained using edge information created by Segment Anything. Our method can be used for image recognition models with various tasks. SA-ICM is also robust to changes in input data, making it effective for a variety of use cases. Additionally, our method provides benefits from a privacy point of view, as it removes human facial information on the encoder's side, thus protecting one's privacy. Furthermore, this LIC model training method can be used to train Neural Representations for Videos (NeRV), which is a video compression model. By training NeRV using edge information created by Segment Anything, it is possible to create a NeRV that is effective for image recognition (SA-NeRV). Experimental results confirm the advantages of SA-ICM, presenting the best performance in image compression for image recognition. We also show that SA-NeRV is superior to ordinary NeRV in video compression for machines. Code is available at https://github.com/final-0/SA-ICM.
Paper Structure (13 sections, 6 equations, 9 figures, 1 table)

This paper contains 13 sections, 6 equations, 9 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Image Coding for Machines (ICM)
  4. Neural Representations for Videos (NeRV)
  5. Proposed Method
  6. SA-ICM
  7. SA-NeRV
  8. Experiments
  9. Experimental Methods for Evaluating SA-ICM
  10. SA-ICM Evaluation Experimental Results
  11. Experimental Methods for Evaluating SA-NeRV
  12. SA-NeRV Evaluation Experimental Results
  13. Conclusion

Figures (9)

  • Figure 1: Overview of image compression process. (a) : LIC model for human vision. (b) : ROI-based approach for ICM. (c) : Task-loss-based approach for ICM. (d) : Region-Learning-based approach for ICM.
  • Figure 2: Examples of the mask image. (a) : Original image. (b) : Mask image in COCO dataset. (c) : Mask image generated using SAM ($\alpha=0.98$). (d) : Mask image generated using SAM ($\alpha=0.93$). (e) : Mask image generated using SAM ($\alpha=0.48$).
  • Figure 3: The proposed training method of the LIC model.
  • Figure 4: Examples of coded images of the COCO2017 dataset. The top line is the input image, the middle line is the coded image by the conventional method of RL-based approach (Object-ICM)c1, and the bottom line is the coded image by the proposed method (SA-ICM).
  • Figure 5: Compression performance in object detection accuracy of YOLOv5. The left figure shows compression performance for COCO, and the right figure shows the same for VisDrone.
  • ...and 4 more figures