Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Neural Image Compression Using Masked Sparse Visual Representation

Wei Jiang, Wei Wang, Yue Chen

TL;DR

This paper tackles neural image compression via Sparse Visual Representation (SVR) by enabling robust, cross-platform transmission of integer codeword indices and addressing the rate–distortion tradeoff. It introduces Masked Adaptive Codebook learning (M-AdaCode), which masks latent-space weights to reduce transmitted data while a decoder-weight refiller reconstructs a rich latent feature using adaptive, semantic-class-dependent codebooks. The method achieves better rate–distortion tradeoffs across a wide range of bitrates on the JPEG-AI dataset, with notable improvements in SSIM and perceptual quality (LPIPS), while controlling bitrate through the masking rate defined by the number of kept codebooks per super-pixel. This work advances practical SVR-based neural image compression by balancing reconstruction fidelity and transmission costs, offering robust performance across heterogeneous hardware platforms and content types.

Abstract

We study neural image compression based on the Sparse Visual Representation (SVR), where images are embedded into a discrete latent space spanned by learned visual codebooks. By sharing codebooks with the decoder, the encoder transfers integer codeword indices that are efficient and cross-platform robust, and the decoder retrieves the embedded latent feature using the indices for reconstruction. Previous SVR-based compression lacks effective mechanism for rate-distortion tradeoffs, where one can only pursue either high reconstruction quality or low transmission bitrate. We propose a Masked Adaptive Codebook learning (M-AdaCode) method that applies masks to the latent feature subspace to balance bitrate and reconstruction quality. A set of semantic-class-dependent basis codebooks are learned, which are weighted combined to generate a rich latent feature for high-quality reconstruction. The combining weights are adaptively derived from each input image, providing fidelity information with additional transmission costs. By masking out unimportant weights in the encoder and recovering them in the decoder, we can trade off reconstruction quality for transmission bits, and the masking rate controls the balance between bitrate and distortion. Experiments over the standard JPEG-AI dataset demonstrate the effectiveness of our M-AdaCode approach.

Neural Image Compression Using Masked Sparse Visual Representation

TL;DR

This paper tackles neural image compression via Sparse Visual Representation (SVR) by enabling robust, cross-platform transmission of integer codeword indices and addressing the rate–distortion tradeoff. It introduces Masked Adaptive Codebook learning (M-AdaCode), which masks latent-space weights to reduce transmitted data while a decoder-weight refiller reconstructs a rich latent feature using adaptive, semantic-class-dependent codebooks. The method achieves better rate–distortion tradeoffs across a wide range of bitrates on the JPEG-AI dataset, with notable improvements in SSIM and perceptual quality (LPIPS), while controlling bitrate through the masking rate defined by the number of kept codebooks per super-pixel. This work advances practical SVR-based neural image compression by balancing reconstruction fidelity and transmission costs, offering robust performance across heterogeneous hardware platforms and content types.

Abstract

We study neural image compression based on the Sparse Visual Representation (SVR), where images are embedded into a discrete latent space spanned by learned visual codebooks. By sharing codebooks with the decoder, the encoder transfers integer codeword indices that are efficient and cross-platform robust, and the decoder retrieves the embedded latent feature using the indices for reconstruction. Previous SVR-based compression lacks effective mechanism for rate-distortion tradeoffs, where one can only pursue either high reconstruction quality or low transmission bitrate. We propose a Masked Adaptive Codebook learning (M-AdaCode) method that applies masks to the latent feature subspace to balance bitrate and reconstruction quality. A set of semantic-class-dependent basis codebooks are learned, which are weighted combined to generate a rich latent feature for high-quality reconstruction. The combining weights are adaptively derived from each input image, providing fidelity information with additional transmission costs. By masking out unimportant weights in the encoder and recovering them in the decoder, we can trade off reconstruction quality for transmission bits, and the masking rate controls the balance between bitrate and distortion. Experiments over the standard JPEG-AI dataset demonstrate the effectiveness of our M-AdaCode approach.
Paper Structure (15 sections, 2 equations, 4 figures, 1 table)

This paper contains 15 sections, 2 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Works
  3. Sparse Visual Representation Learning
  4. Neural Image Compression
  5. SVR-based Compression
  6. Masked Image Modeling
  7. Approach
  8. Weight Masking and Refilling
  9. Single Codebook Setup
  10. Training Process
  11. Experiments
  12. Reconstruction performance
  13. Ablation Study
  14. More Discussions
  15. Conclusion

Figures (4)

  • Figure 1: Different neural image compression frameworks.
  • Figure 2: Quantitative comparison with SOTA SVR-based compression methods. PSNR/SSIM: the higher, the better. LPIPS: the lower, the better. Previous MAGE MAGE and AdaCode AdaCode operate with very low or high bitrates. M-Adacode provides better rate-distortion tradeoffs over a range of bitrates.
  • Figure 3: Reconstruction examples. Numbers under each result are "LPIPS$|$PSNR$|$SSIM". "M-AdaCode 1-codebook" and "M-AdaCode 2-codebook" are M-AdaCode using 1 codebook or 2 codebooks per super-pixel, respectively.
  • Figure 4: Ablation study: performance without weight filler and performance without single codebook setting. Weight filler can largely improve pixel-level distortion, and the single codebook setting can reduce bitrate without hurting distortion.