GRACE: Loss-Resilient Real-Time Video through Neural Codecs

TL;DR

Grace tackles the challenge of maintaining QoE in real-time video over lossy, high-latency networks without retransmission. It introduces data-scalable coding by jointly training a neural video encoder and decoder under simulated packet losses, enabling frames to be decoded from any non-empty subset of packets and ensuring graceful quality degradation as losses rise. Empirically, Grace achieves near-H.265 quality with no loss, substantially reduces undecodable frames and stalls compared to FEC and error-concealment baselines, and receives higher user ratings in crowdsourced MOS studies. The work demonstrates end-to-end feasibility with real-network traces and mobile hardware, offering a practical pathway to robust real-time video in unreliable networks.

Abstract

In real-time video communication, retransmitting lost packets over high-latency networks is not viable due to strict latency requirements. To counter packet losses without retransmission, two primary strategies are employed -- encoder-based forward error correction (FEC) and decoder-based error concealment. The former encodes data with redundancy before transmission, yet determining the optimal redundancy level in advance proves challenging. The latter reconstructs video from partially received frames, but dividing a frame into independently coded partitions inherently compromises compression efficiency, and the lost information cannot be effectively recovered by the decoder without adapting the encoder. We present a loss-resilient real-time video system called GRACE, which preserves the user's quality of experience (QoE) across a wide range of packet losses through a new neural video codec. Central to GRACE's enhanced loss resilience is its joint training of the neural encoder and decoder under a spectrum of simulated packet losses. In lossless scenarios, GRACE achieves video quality on par with conventional codecs (e.g., H.265). As the loss rate escalates, GRACE exhibits a more graceful, less pronounced decline in quality, consistently outperforming other loss-resilient schemes. Through extensive evaluation on various videos and real network traces, we demonstrate that GRACE reduces undecodable frames by 95% and stall duration by 90% compared with FEC, while markedly boosting video quality over error concealment methods. In a user study with 240 crowdsourced participants and 960 subjective ratings, GRACE registers a 38% higher mean opinion score (MOS) than other baselines.

Figures (29)

• Figure 1: Illustration of the video quality achieved by different loss-resilient schemes, operating under the same bandwidth budget, across varying packet loss rates. Actual experimental results are shown in Figure \ref{['fig:loss_by_dataset']}.
• Figure 2: A typical workflow of video frame encoding.
• Figure 3: Workflow of Grace's neural video codec.
• Figure 4: Unlike traditional NVC training that assumes no data loss between the encoder and decoder, Grace applies "random masking"---setting a fraction of randomly selected elements to zeros---to the encoder's output.
• Figure 5: Grace's reversible randomized packetization. The tensor elements mapped to a lost packet will be set to zeros.