Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Conditional Latent Coding with Learnable Synthesized Reference for Deep Image Compression

Siqi Wu, Yinda Chen, Dong Liu, Zhihai He

TL;DR

This paper tackles efficient deep image compression by exploiting external, dynamically synthesized references in the latent domain. It introduces Conditional Latent Coding (CLC), which builds a universal feature dictionary from large image collections and learns to fuse multiple dictionary features into a conditioning latent that guides encoding and decoding. The method combines a fast dictionary search with CLM/CLS modules for feature matching and synthesis, augmented by a hyperprior and slice-based autoregressive entropy model, and it is complemented by a perturbation analysis that bounds feature-retrieval errors under auxiliary information. Empirical results show substantial rate-distortion gains (up to 1.2 dB) at very low bitrates with modest bitrate overhead (~0.5%), validating the approach's robustness and practical impact for scalable image compression. The work also provides code and outlines potential extensions to broader vision tasks and multi-modal data scenarios.

Abstract

In this paper, we study how to synthesize a dynamic reference from an external dictionary to perform conditional coding of the input image in the latent domain and how to learn the conditional latent synthesis and coding modules in an end-to-end manner. Our approach begins by constructing a universal image feature dictionary using a multi-stage approach involving modified spatial pyramid pooling, dimension reduction, and multi-scale feature clustering. For each input image, we learn to synthesize a conditioning latent by selecting and synthesizing relevant features from the dictionary, which significantly enhances the model's capability in capturing and exploring image source correlation. This conditional latent synthesis involves a correlation-based feature matching and alignment strategy, comprising a Conditional Latent Matching (CLM) module and a Conditional Latent Synthesis (CLS) module. The synthesized latent is then used to guide the encoding process, allowing for more efficient compression by exploiting the correlation between the input image and the reference dictionary. According to our theoretical analysis, the proposed conditional latent coding (CLC) method is robust to perturbations in the external dictionary samples and the selected conditioning latent, with an error bound that scales logarithmically with the dictionary size, ensuring stability even with large and diverse dictionaries. Experimental results on benchmark datasets show that our new method improves the coding performance by a large margin (up to 1.2 dB) with a very small overhead of approximately 0.5\% bits per pixel. Our code is publicly available at https://github.com/ydchen0806/CLC.

Conditional Latent Coding with Learnable Synthesized Reference for Deep Image Compression

TL;DR

This paper tackles efficient deep image compression by exploiting external, dynamically synthesized references in the latent domain. It introduces Conditional Latent Coding (CLC), which builds a universal feature dictionary from large image collections and learns to fuse multiple dictionary features into a conditioning latent that guides encoding and decoding. The method combines a fast dictionary search with CLM/CLS modules for feature matching and synthesis, augmented by a hyperprior and slice-based autoregressive entropy model, and it is complemented by a perturbation analysis that bounds feature-retrieval errors under auxiliary information. Empirical results show substantial rate-distortion gains (up to 1.2 dB) at very low bitrates with modest bitrate overhead (~0.5%), validating the approach's robustness and practical impact for scalable image compression. The work also provides code and outlines potential extensions to broader vision tasks and multi-modal data scenarios.

Abstract

In this paper, we study how to synthesize a dynamic reference from an external dictionary to perform conditional coding of the input image in the latent domain and how to learn the conditional latent synthesis and coding modules in an end-to-end manner. Our approach begins by constructing a universal image feature dictionary using a multi-stage approach involving modified spatial pyramid pooling, dimension reduction, and multi-scale feature clustering. For each input image, we learn to synthesize a conditioning latent by selecting and synthesizing relevant features from the dictionary, which significantly enhances the model's capability in capturing and exploring image source correlation. This conditional latent synthesis involves a correlation-based feature matching and alignment strategy, comprising a Conditional Latent Matching (CLM) module and a Conditional Latent Synthesis (CLS) module. The synthesized latent is then used to guide the encoding process, allowing for more efficient compression by exploiting the correlation between the input image and the reference dictionary. According to our theoretical analysis, the proposed conditional latent coding (CLC) method is robust to perturbations in the external dictionary samples and the selected conditioning latent, with an error bound that scales logarithmically with the dictionary size, ensuring stability even with large and diverse dictionaries. Experimental results on benchmark datasets show that our new method improves the coding performance by a large margin (up to 1.2 dB) with a very small overhead of approximately 0.5\% bits per pixel. Our code is publicly available at https://github.com/ydchen0806/CLC.

Paper Structure

This paper contains 32 sections, 3 theorems, 31 equations, 8 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work and Unique Contributions
  3. The Proposed CLC Method
  4. Method Overview
  5. Constructing the Support Dictionary
  6. (1) Constructing the reference feature dictionary.
  7. (2) Fast and efficient dictionary matching.
  8. Conditional Latent Synthesis and Coding
  9. (1) Feature Matching and Alignment.
  10. (2) Conditional Latent Synthesis.
  11. (3) Entropy Coding and Hyperprior.
  12. (4) Decoding and Optimization.
  13. Theoretical Perturbation Analysis
  14. Experimental Results
  15. Experimental Settings
  16. ...and 17 more sections

Key Result

Theorem 1

For any $\delta > 0$, with probability at least $1-\delta$: where $C > 0$ is a constant, $p$ is the proportion of irrelevant parts in the auxiliary image, $n$ is the number of training samples, $r(\Sigma_\xi)$ is the effective rank of the noise covariance matrix, and $\hat{G}_1$ is the estimated encoder for the original image.

Figures (8)

  • Figure 1: Overview of the proposed Conditional Latent Coding (CLC) framework.
  • Figure 2: Universal Feature Dictionary Construction. (a) Dictionary Generation using diverse images $\mathcal{R}$ to create initial $\mathcal{D}$. (b) Reference Retrieval for querying and updating dictionary with inputs $x$ and $x'$. (c) Examples of reference candidates $X_r^M$ retrieved from the dictionary.
  • Figure 3: The detail of our proposed CLM and CLS module.
  • Figure 4: The rate-distortion performance comparison of different methods.
  • Figure 5: Image reconstruction results at around 0.1 bpp. From left to right: Raw inputs, reference images, reconstructed images. Red and blue boxes highlight specific areas of improvement.
  • ...and 3 more figures

Theorems & Definitions (10)

  • Theorem 1
  • Lemma 1
  • proof
  • Theorem 2
  • proof
  • Remark 1
  • Remark 2
  • Remark 3
  • Remark 4
  • Remark 5