Robust Multiple Description Neural Video Codec with Masked Transformer for Dynamic and Noisy Networks

TL;DR

This work tackles robust video delivery over dynamic and noisy networks by introducing NeuralMDC, a neural multiple-description codec. It tokenizes frames into latent representations, splits them into multiple correlated descriptions, and uses a bidirectional masked Transformer to model $P(y_t|y_{t-1},y_{t-2})$ for independent entropy coding, while inferring lost tokens from context. The approach achieves state-of-the-art loss resilience with competitive rate-distortion, offering substantial bitrate savings (e.g., up to $76.88\%$) over prior neural MDC baselines and strong performance under RTX and time-varying RTT conditions. This has practical implications for 5G and beyond, enabling reliable, multi-path video delivery with reduced latency and improved quality under packet losses.

Abstract

Multiple Description Coding (MDC) is a promising error-resilient source coding method that is particularly suitable for dynamic networks with multiple (yet noisy and unreliable) paths. However, conventional MDC video codecs suffer from cumbersome architectures, poor scalability, limited loss resilience, and lower compression efficiency. As a result, MDC has never been widely adopted. Inspired by the potential of neural video codecs, this paper rethinks MDC design. We propose a novel MDC video codec, NeuralMDC, demonstrating how bidirectional transformers trained for masked token prediction can vastly simplify the design of MDC video codec. To compress a video, NeuralMDC starts by tokenizing each frame into its latent representation and then splits the latent tokens to create multiple descriptions containing correlated information. Instead of using motion prediction and warping operations, NeuralMDC trains a bidirectional masked transformer to model the spatial-temporal dependencies of latent representations and predict the distribution of the current representation based on the past. The predicted distribution is used to independently entropy code each description and infer any potentially lost tokens. Extensive experiments demonstrate NeuralMDC achieves state-of-the-art loss resilience with minimal sacrifices in compression efficiency, significantly outperforming the best existing residual-coding-based error-resilient neural video codec.

Figures (13)

• Figure 1: Overview of NeuralMDC codec: an example of generating 4 descriptions.
• Figure 2: Sorted channel maps with the top-4 largest energy. Left 1: the original frame. The strongest activation is concentrated in the first channel map (left 2), while the remaining channels become increasingly sparse.
• Figure 3: Latent representation separation example: the $1/4$ description.
• Figure 4: Impact of losses of motion vs. residuals on the video quality of Grace cheng2024grace, a loss-resilient residual-coding codec. Figure labels indicate which source information is corrupted while the other is fully received.
• Figure 5: Overview of the masked transformer entropy model. During training, the model learns to predict the distributions of masked tokens. At inference, the model begins by predicting all masked tokens and then follows the QLDS masking schedule to keep a portion of predicted tokens as input for the next prediction iteration. This process continues until all tokens are uncovered.