Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MLIC: Multi-Reference Entropy Model for Learned Image Compression

Wei Jiang, Jiayu Yang, Yongqi Zhai, Peirong Ning, Feng Gao, Ronggang Wang

TL;DR

This work proposes the Multi-Reference Entropy Model (MEM) and the advanced version, MEM+.

Abstract

Recently, learned image compression has achieved remarkable performance. The entropy model, which estimates the distribution of the latent representation, plays a crucial role in boosting rate-distortion performance. However, most entropy models only capture correlations in one dimension, while the latent representation contain channel-wise, local spatial, and global spatial correlations. To tackle this issue, we propose the Multi-Reference Entropy Model (MEM) and the advanced version, MEM$^+$. These models capture the different types of correlations present in latent representation. Specifically, We first divide the latent representation into slices. When decoding the current slice, we use previously decoded slices as context and employ the attention map of the previously decoded slice to predict global correlations in the current slice. To capture local contexts, we introduce two enhanced checkerboard context capturing techniques that avoids performance degradation. Based on MEM and MEM$^+$, we propose image compression models MLIC and MLIC$^+$. Extensive experimental evaluations demonstrate that our MLIC and MLIC$^+$ models achieve state-of-the-art performance, reducing BD-rate by $8.05\%$ and $11.39\%$ on the Kodak dataset compared to VTM-17.0 when measured in PSNR. Our code is available at https://github.com/JiangWeibeta/MLIC.

MLIC: Multi-Reference Entropy Model for Learned Image Compression

TL;DR

This work proposes the Multi-Reference Entropy Model (MEM) and the advanced version, MEM+.

Abstract

Recently, learned image compression has achieved remarkable performance. The entropy model, which estimates the distribution of the latent representation, plays a crucial role in boosting rate-distortion performance. However, most entropy models only capture correlations in one dimension, while the latent representation contain channel-wise, local spatial, and global spatial correlations. To tackle this issue, we propose the Multi-Reference Entropy Model (MEM) and the advanced version, MEM. These models capture the different types of correlations present in latent representation. Specifically, We first divide the latent representation into slices. When decoding the current slice, we use previously decoded slices as context and employ the attention map of the previously decoded slice to predict global correlations in the current slice. To capture local contexts, we introduce two enhanced checkerboard context capturing techniques that avoids performance degradation. Based on MEM and MEM, we propose image compression models MLIC and MLIC. Extensive experimental evaluations demonstrate that our MLIC and MLIC models achieve state-of-the-art performance, reducing BD-rate by and on the Kodak dataset compared to VTM-17.0 when measured in PSNR. Our code is available at https://github.com/JiangWeibeta/MLIC.
Paper Structure (32 sections, 17 equations, 26 figures, 5 tables)

This paper contains 32 sections, 17 equations, 26 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Learned Lossy Image Compression
  4. Context-based Entropy Model
  5. Method
  6. Motivation
  7. Overview
  8. MLIC and MLIC$^+$
  9. MEM and MEM$^+$
  10. Channel-wise Context Module
  11. Enhanced Checkerboard Context Module
  12. Stacked Checkerboard Context Module
  13. Shifted-Window-based Checkerboard Attention
  14. Global Spatial Context Module
  15. Intra-Slice Global Context Module
  16. ...and 17 more sections

Figures (26)

  • Figure 1: Context Model Comparison. Our proposed Multi-reference Entropy Model MEM and MEM$^+$ can capture correlations in the local spatial, global spatial, and channel dimensions.
  • Figure 2: Visualization of channels of latent representation of Kodim19 extracted by Cheng'20 DBLP:conf/cvpr/ChengSTK20(optimized for MSE, $\lambda=0.0483$).
  • Figure 3: Heatmap of spatial cosine similarity of latent representation of Kodim19 extracted by Cheng'20 DBLP:conf/cvpr/ChengSTK20 (optimized for MSE, $\lambda=0.0483$).
  • Figure 4: The overall architecture of MLIC and MLIC$^+$. $\downarrow$ means down-sampling. $\uparrow$ means up-sampling. / means stride equals $1$. Red line is the dataflow during decoding. Please refer to Table \ref{['tab:notation']} for the explanations of other notations.
  • Figure 5: Multi-Reference Entropy Model MEM and MEM$^+$. The figure illustrates the process of decoding a slice $\hat{\boldsymbol{y}}^i$.
  • ...and 21 more figures