BiECVC: Gated Diversification of Bidirectional Contexts for Learned Video Compression

TL;DR

BiECVC tackles the underexplored area of learned bidirectional video compression by diversifying bidirectional contexts into local and non-local categories and by employing adaptive Bidirectional Context Gating. It introduces a feature cache to accelerate inference and uses linear attention to manage non-local dependencies efficiently. The method achieves state-of-the-art rate-distortion performance, surpassing VTM 13.2 RA across standard test datasets, with BD-rate reductions of $-13.4\%$ and $-15.7\%$ under IP32 with intra periods of 32 and 64, respectively, marking the first learned codec to beat VTM RA on all tested sets. These results demonstrate the practical impact of diversified contexts and data-driven gating for improving compression efficiency in learned video codecs, though real-time deployment remains future work.

Abstract

Recent forward prediction-based learned video compression (LVC) methods have achieved impressive results, even surpassing VVC reference software VTM under the Low Delay B (LDB) configuration. In contrast, learned bidirectional video compression (BVC) remains underexplored and still lags behind its forward-only counterparts. This performance gap is mainly due to the limited ability to extract diverse and accurate contexts: most existing BVCs primarily exploit temporal motion while neglecting non-local correlations across frames. Moreover, they lack the adaptability to dynamically suppress harmful contexts arising from fast motion or occlusion. To tackle these challenges, we propose BiECVC, a BVC framework that incorporates diversified local and non-local context modeling along with adaptive context gating. For local context enhancement, BiECVC reuses high-quality features from lower layers and aligns them using decoded motion vectors without introducing extra motion overhead. To model non-local dependencies efficiently, we adopt a linear attention mechanism that balances performance and complexity. To further mitigate the impact of inaccurate context prediction, we introduce Bidirectional Context Gating, inspired by data-dependent decay in recent autoregressive language models, to dynamically filter contextual information based on conditional coding results. Extensive experiments demonstrate that BiECVC achieves state-of-the-art performance, reducing the bit-rate by 13.4% and 15.7% compared to VTM 13.2 under the Random Access (RA) configuration with intra periods of 32 and 64, respectively. To our knowledge, BiECVC is the first learned video codec to surpass VTM 13.2 RA across all standard test datasets.

Figures (17)

• Figure 1: Overall framework of BiECVC. The quantization parameter (qp) controls variable-rate compression. Local contexts $\{\hat{\boldsymbol{L}}^{\ell}_t\}_{\ell=0}^2$ are shared between the encoder and decoder. The non-local contexts, $\{{\boldsymbol{G}}^{\ell}_t\}_{\ell=0}^2$ and $\{\hat{\boldsymbol{G}}^{\ell}_t\}_{\ell=0}^2$, are extracted separately at the encoder and decoder. The final contexts, ${\boldsymbol{C}_t^{\ell}}$ and ${\hat{\boldsymbol{C}}_t^{\ell}}$, are used at the contextual encoder and decoder, respectively. AE and AD are arithmetic encoding and decoding, respectively.
• Figure 2:
• Figure 3: Illustrations of the proposed Bidirectional Local Context Diversification (BLCD) and Bidirectional Non-Local Context Diversification (BNLCD) and Bidirectional Context Gating (BCG). The process is conditional coding $\boldsymbol{y}_t^2$ at the encoder side. The decoding process mirrors the encoding process.
• Figure 4: Visualization of local context (with motion trajectory) and non-local context (no motion trajectory).
• Figure 5: Visualization of predicted errors in optical flow-based warping. The frames are from "Jockey" sequence in UVG mercat2020uvg dataset.