Unified Coding for Both Human Perception and Generalized Machine Analytics with CLIP Supervision

TL;DR

UG-ICM tackles the generalization gap in image coding by learning a single latent representation $\hat{y}$ and employing a Preference Conditional Decoding Module to produce human- or machine-preferred reconstructions from one bitstream. It integrates Multi-Scale CLIP supervision to impose global, local, and instance-level semantic alignment, enabling zero-task supervision and better generalization to unseen analytics tasks. A two-stage training regime balances rate, distortion, perceptual quality, and analytics while remaining task-agnostic. Experimental results on two LIC backbones show consistent improvements in unseen classification, detection, and segmentation tasks with maintained perceptual quality, outperforming several state-of-the-art ICM methods.

Abstract

The image compression model has long struggled with adaptability and generalization, as the decoded bitstream typically serves only human or machine needs and fails to preserve information for unseen visual tasks. Therefore, this paper innovatively introduces supervision obtained from multimodal pre-training models and incorporates adaptive multi-objective optimization tailored to support both human visual perception and machine vision simultaneously with a single bitstream, denoted as Unified and Generalized Image Coding for Machine (UG-ICM). Specifically, to get rid of the reliance between compression models with downstream task supervision, we introduce Contrastive Language-Image Pre-training (CLIP) models into the training constraint for improved generalization. Global-to-instance-wise CLIP supervision is applied to help obtain hierarchical semantics that make models more generalizable for the tasks relying on the information of different granularity. Furthermore, for supporting both human and machine visions with only a unifying bitstream, we incorporate a conditional decoding strategy that takes as conditions human or machine preferences, enabling the bitstream to be decoded into different versions for corresponding preferences. As such, our proposed UG-ICM is fully trained in a self-supervised manner, i.e., without awareness of any specific downstream models and tasks. The extensive experiments have shown that the proposed UG-ICM is capable of achieving remarkable improvements in various unseen machine analytics tasks, while simultaneously providing perceptually satisfying images.

Figures (10)

• Figure 1: (a) Most existing ICM models rely on task-specific supervision and fall short of generalization capacity in unseen scenarios. (b) $z$ and $\hat{z}$ denote the latent representations satisfying downstream tasks from the original image $x$ and compressed image $\hat{x}$, respectively.
• Figure 2: (a) Compressing pipeline of the proposed UG-ICM. (b) Details of the proposed PCDM. (c) Modules and loss terms involved in the multi-stage training process.
• Figure 3: Empirical analysis on generalization of taking CLIP as supervision constraint.
• Figure 4: Illustration of the proposed Multi-Scale CLIP loss.
• Figure 5: Machine analytics performance comparisons between the proposed UG-ICM and the backbone compression networks.