LL-ICM: Image Compression for Low-level Machine Vision via Large Vision-Language Model

TL;DR

This paper proposes a pioneered ICM framework for LL machine vision tasks, namely LL-ICM, and establishes a solid benchmark to evaluate LL-ICM, which shows that LL-ICM can achieve 22.65% BD-rate reductions over the state-of-the-art methods.

Abstract

Image Compression for Machines (ICM) aims to compress images for machine vision tasks rather than human viewing. Current works predominantly concentrate on high-level tasks like object detection and semantic segmentation. However, the quality of original images is usually not guaranteed in the real world, leading to even worse perceptual quality or downstream task performance after compression. Low-level (LL) machine vision models, like image restoration models, can help improve such quality, and thereby their compression requirements should also be considered. In this paper, we propose a pioneered ICM framework for LL machine vision tasks, namely LL-ICM. By jointly optimizing compression and LL tasks, the proposed LL-ICM not only enriches its encoding ability in generalizing to versatile LL tasks but also optimizes the processing ability of down-stream LL task models, achieving mutual adaptation for image codecs and LL task models. Furthermore, we integrate large-scale vision-language models into the LL-ICM framework to generate more universal and distortion-robust feature embeddings for LL vision tasks. Therefore, one LL-ICM codec can generalize to multiple tasks. We establish a solid benchmark to evaluate LL-ICM, which includes extensive objective experiments by using both full and no-reference image quality assessments. Experimental results show that LL-ICM can achieve 22.65% BD-rate reductions over the state-of-the-art methods.

Figures (6)

• Figure 1: ICM frameworks from the simplest to the proposed. $\mathbf{X}$ is the original image, $\mathbf{\hat{X}}$ is the compressed image, and $\mathbf{\hat{X}_H}$ is the enhanced image by downstream task model. $\mathbf{Cls.A}$ and $\mathbf{Seg.B}$ are the results of HL tasks A and B.
• Figure 2: Difference between RPO and RDO Definition: RDO distortion measures the error of compressed image $\mathbf{\hat{X}}$ and the original image $\mathbf{X}$, while RPO measures the error of the generated enhanced image $\mathbf{\hat{X}_H}$ and the high-quality ideal image $\mathbf{X_{ideal}}$.
• Figure 3: Overview of the LL-ICM framework: $\mathbf{\hat{X}}$ is the compressed image. A pre-trained CLIP model extracts the generalized feature $\mathbf{F}$ from $\mathbf{\hat{X}}$. After the DA-CLIP encoder $\mathbf{E}$, the encoded $\mathbf{F}$ can be used for multiple LL tasks and reconstruct the enhanced image $\mathbf{\hat{X}_H}$.
• Figure 4: RP performance on 6 LL tasks: Our LL-ICM framework significantly outperforms all existing codecs. RP performance on (a)-(d) are the monotonic cases in which the perception quality increases as the bpp increases. The (e) and (f) are abnormal cases as the perception quality does not follow the regular law as (a)-(d).
• Figure 5: An abnormal result on denoising: the noise information is more in higher bpp image compared to lower bpp image.