Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PerCo (SD): Open Perceptual Compression

Nikolai Körber, Eduard Kromer, Andreas Siebert, Sascha Hauke, Daniel Mueller-Gritschneder, Björn Schuller

TL;DR

This work tackles ultra-low bit-rate perceptual image compression by introducing PerCo (SD), an open implementation based on Stable Diffusion v2.1 that aims to match the performance of the proprietary PerCo. It encodes local and global conditioning through vector-quantized hyper-latents and BLIP-2 captions, conditioning a latent diffusion model for reconstruction. The authors provide a thorough comparison on Kodak and MSCOCO-30k, showing competitive perceptual metrics at very low bitrates while incurring higher distortion due to model capacity differences, and they discuss design choices to make the approach open and reproducible. The release of code and models is highlighted to foster broader research and future improvements in open perceptual compression.

Abstract

We introduce PerCo (SD), a perceptual image compression method based on Stable Diffusion v2.1, targeting the ultra-low bit range. PerCo (SD) serves as an open and competitive alternative to the state-of-the-art method PerCo, which relies on a proprietary variant of GLIDE and remains closed to the public. In this work, we review the theoretical foundations, discuss key engineering decisions in adapting PerCo to the Stable Diffusion ecosystem, and provide a comprehensive comparison, both quantitatively and qualitatively. On the MSCOCO-30k dataset, PerCo (SD) demonstrates improved perceptual characteristics at the cost of higher distortion. We partly attribute this gap to the different model capacities being used (866M vs. 1.4B). We hope our work contributes to a deeper understanding of the underlying mechanisms and paves the way for future advancements in the field. Code and trained models will be released at https://github.com/Nikolai10/PerCo.

PerCo (SD): Open Perceptual Compression

TL;DR

This work tackles ultra-low bit-rate perceptual image compression by introducing PerCo (SD), an open implementation based on Stable Diffusion v2.1 that aims to match the performance of the proprietary PerCo. It encodes local and global conditioning through vector-quantized hyper-latents and BLIP-2 captions, conditioning a latent diffusion model for reconstruction. The authors provide a thorough comparison on Kodak and MSCOCO-30k, showing competitive perceptual metrics at very low bitrates while incurring higher distortion due to model capacity differences, and they discuss design choices to make the approach open and reproducible. The release of code and models is highlighted to foster broader research and future improvements in open perceptual compression.

Abstract

We introduce PerCo (SD), a perceptual image compression method based on Stable Diffusion v2.1, targeting the ultra-low bit range. PerCo (SD) serves as an open and competitive alternative to the state-of-the-art method PerCo, which relies on a proprietary variant of GLIDE and remains closed to the public. In this work, we review the theoretical foundations, discuss key engineering decisions in adapting PerCo to the Stable Diffusion ecosystem, and provide a comprehensive comparison, both quantitatively and qualitatively. On the MSCOCO-30k dataset, PerCo (SD) demonstrates improved perceptual characteristics at the cost of higher distortion. We partly attribute this gap to the different model capacities being used (866M vs. 1.4B). We hope our work contributes to a deeper understanding of the underlying mechanisms and paves the way for future advancements in the field. Code and trained models will be released at https://github.com/Nikolai10/PerCo.
Paper Structure (13 sections, 5 equations, 13 figures, 1 table)

This paper contains 13 sections, 5 equations, 13 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Perceptual compression
  4. Model overview
  5. PerCo (SD)
  6. Experimental results
  7. Main results
  8. Ablations/ further results
  9. Related work
  10. Conclusion
  11. Appendix / supplemental material
  12. Additional quantitative results
  13. Additional visual results

Figures (13)

  • Figure 1: Visual comparison of PerCo (SD) to PICS lei2023text+sketch, VTM-20.0, the state-of-the-art non-learned image codec, and PerCo careil2024towards. Notably, PerCo and PerCo (SD) achieve an order of magnitude lower bits per pixel (bpp) compared to competing methods. Best viewed electronically.
  • Figure 2: PerCo model overview, adopted from careil2024towards. During training, the hyper-encoder, codebook, and diffusion model are trained, whereas all other components are fixed.
  • Figure 3: Quantitative comparison: PerCo (official) vs. PerCo (SD)
  • Figure 4: Quantitative comparison on the MSCOCO-30k dataset: PerCo (official) vs. PerCo (SD). We have not tried to tune our model towards better PSNR scores, as these low-level distortion metrics are known to be less meaningful for low rates careil2024towards.
  • Figure 5: Quantitative comparison on the Kodak dataset: PerCo (official) vs. PerCo (SD). We further show another model configuration based on the EulerAncestralDiscreteScheduler, which we found to produce consistently lower distortion at the cost of, however, slightly decreased perceptual characteristics. Note that the PerCo (SD) performance is bounded by the auto-encoder.
  • ...and 8 more figures