ResiComp: Loss-Resilient Image Compression via Dual-Functional Masked Visual Token Modeling

TL;DR

ResiComp addresses the vulnerability of neural image codecs to packet loss in real-time communications by unifying entropy modeling and latent space packet loss concealment within a single dual-functional MVTM Transformer. By partitioning latents into packet-aligned slices and training with masked token prediction, it achieves adjustable efficiency-resilience trade-offs through context-mode scheduling and QLDS-based slice partitioning. The approach demonstrates stronger resilience than VTM+FEC across diverse packet-loss scenarios while maintaining competitive compression efficiency, and supports progressive decoding and robust operation under variable network conditions. The work advances practical NIC deployment for RTC and lays groundwork for deeper integration with inter-frame coding and physical-layer techniques.

Abstract

Recent advancements in neural image codecs (NICs) are of significant compression performance, but limited attention has been paid to their error resilience. These resulting NICs tend to be sensitive to packet losses, which are prevalent in real-time communications. In this paper, we investigate how to elevate the resilience ability of NICs to combat packet losses. We propose ResiComp, a pioneering neural image compression framework with feature-domain packet loss concealment (PLC). Motivated by the inherent consistency between generation and compression, we advocate merging the tasks of entropy modeling and PLC into a unified framework focused on latent space context modeling. To this end, we take inspiration from the impressive generative capabilities of large language models (LLMs), particularly the recent advances of masked visual token modeling (MVTM). During training, we integrate MVTM to mirror the effects of packet loss, enabling a dual-functional Transformer to restore the masked latents by predicting their missing values and conditional probability mass functions. Our ResiComp jointly optimizes compression efficiency and loss resilience. Moreover, ResiComp provides flexible coding modes, allowing for explicitly adjusting the efficiency-resilience trade-off in response to varying Internet or wireless network conditions. Extensive experiments demonstrate that ResiComp can significantly enhance the NIC's resilience against packet losses, while exhibits a worthy trade-off between compression efficiency and packet loss resilience.

Figures (20)

• Figure 1: Efficiency-resilience trade-off on Kodak. Higher positions in the top-right corner indicate better performance. The resilience performance of some codecs is marked as "N/A," indicating that due to frequent decoding failures, their average PSNR falls below 25 dB. The detailed description will be presented in Section \ref{['resilience-comparison']}.
• Figure 2: Overview of ResiComp. The encoder maps the input image into quantized image tokens. These tokens are divided into multiple slices, and each slice is entropy encoded and packetized into a packet. A bi-directional Transformer is employed with a density head for iterative entropy modeling. Some packets may be lost during transmission, and the affected image tokens are replaced with special mask tokens. We predict the masked tokens using the same Transformer, with a PLC head for packet loss concealment, and reconstruct the input image from the inpainted tokens using a decoder. The detailed loss-resilient image compression process is presented in Sec. \ref{['sec:loss-resilient-image-compression']}, with the workflows of the ResiComp sender and receiver outlined in Algorithm \ref{['algorithm_sender']} and Algorithm \ref{['algorithm_receiver']}. The token decoding progress at the receiver is further explained in Fig. \ref{['Fig_iterative_decoding']}.
• Figure 3: The training pipeline of ResiComp. We randomly sample a masking ratio from 0 to 1 and mask out tokens with $\mathcal{M}$. A bi-directional Transformer models the spatial dependencies among tokens by 1) restoring the values of masked tokens with PLC head; 2) predicting the parametric distributions of masked tokens with density head. The final loss function is the weighted sum of three terms, 1) rate loss: estimated bitrate in masking locations; 2) reconstruction loss: the reconstruct distortion from quantized tokens $\hat{\bm{y}}$; 3) resilient loss: the distortion from reconstructed tokens $\check{\bm{y}}$.
• Figure 4: Top left: original image (kodim01.png) and a cropped patch. Top right: Three slices partitioned via QLDS location schedules with $L=10$ and $\beta=1.0$. It can be observed that all points in a slice are far from each other. We conduct latent PLC from partial token slices (missing values are filled with mask tokens) and decode them using the same model. Bottom (left to right): reconstruct patches decoded from all mask tokens, slice 1 only, slice 1 and 2, slice 1 to 3, respectively.
• Figure 5: Visualization of mask token prediction. It shows our Transformer enables flexible density predictions for $\hat{y}$, with accuracy improving as the number of context slices increases. The location of $\hat{y}$ is highlighted in Fig. \ref{['Fig_mask_token_prediction_a']}, and we visualize the channel with the highest entropy.