Neural Stereo Video Compression with Hybrid Disparity Compensation

TL;DR

This work tackles stereo video compression by addressing cross-view redundancy through a Hybrid Disparity Compensation (HDC) that fuses explicit pixel displacement priors with implicit cross-attention. The authors integrate HDC into a neural stereo video codec built on a DCVC-TCM backbone, introducing HDC-FER for cross-view feature extraction/reconstruction and HDC-EM for cross-view entropy modeling. The framework achieves state-of-the-art rate-distortion performance on automotive benchmarks (Cityscapes, KITTI) and general multi-view scenes (Nagoya), while maintaining competitive computational efficiency. These results demonstrate the effectiveness of a hybrid explicit-implicit disparity strategy for NSVC and suggest strong potential for broader multi-view applications. The approach offers practical impact by enabling more efficient storage and transmission of stereo content in real-world systems such as autonomous driving and VR/AR deployments.

Abstract

Disparity compensation represents the primary strategy in stereo video compression (SVC) for exploiting cross-view redundancy. These mechanisms can be broadly categorized into two types: one that employs explicit horizontal shifting, and another that utilizes an implicit cross-attention mechanism to reduce cross-view disparity redundancy. In this work, we propose a hybrid disparity compensation (HDC) strategy that leverages explicit pixel displacement as a robust prior feature to simplify optimization and perform implicit cross-attention mechanisms for subsequent warping operations, thereby capturing a broader range of disparity information. Specifically, HDC first computes a similarity map by fusing the horizontally shifted cross-view features to capture pixel displacement information. This similarity map is then normalized into an "explicit pixel-wise attention score" to perform the cross-attention mechanism, implicitly aligning features from one view to another. Building upon HDC, we introduce a novel end-to-end optimized neural stereo video compression framework, which integrates HDC-based modules into key coding operations, including cross-view feature extraction and reconstruction (HDC-FER) and cross-view entropy modeling (HDC-EM). Extensive experiments on SVC benchmarks, including KITTI 2012, KITTI 2015, and Nagoya, which cover both autonomous driving and general scenes, demonstrate that our framework outperforms both neural and traditional SVC methodologies.

Figures (8)

• Figure 1: Left: Overview of our Neural Stereo Video Compression (NSVC) framework. At time step $t$, the input stereo frame pair $\{\bm{x}_{t}^L, \bm{x}_{t}^R\}$ is compressed and reconstructed into $\{\bm{\hat{x}}_{t}^L,\bm{\hat{x}}_{t}^R\}$ using five key coding components. This process is conditioned on the previously reconstructed frame pair $\{\bm{\hat{x}}_{t-1}^L, \bm{\hat{x}}_{t-1}^R\}$ and the corresponding feature pair $\{\bm{F}_{t-1}^L, \bm{F}_{t-1}^R\}$ from time step $t-1$ as in established neural video compression (NVC) methods sheng2022tcm. Middle: In the two key components, Motion Compression and Context Compression, we apply and extend our Hybrid Disparity Compensation (HDC) strategy. Specifically, HDC-FER mechanisms are used within their encoders and decoders to enhance the intermediate feature pair $\{\bm{K}_{t}^L, \bm{K}_{t}^R\}$, while HDC-EM mechanisms are used within their entropy model to improve the quantized latent feature pair $\{\bm{\hat{y}}_{t}^L, \bm{\hat{y}}_{t}^R\}$. Right: High-level syntax of the proposed HDC strategy. We first apply explicit pixel displacement to construct a similarity map, which is then used in a cross-attention mechanism to implicitly align features across views.
• Figure 2: The proposed Hybrid Disparity Compensation Module for Feature Extraction and Reconstruction (HDC-FER). We input the intermediate feature pair $\{\bm{K}^L, \bm{K}^R\}$ produced by the encoder and decoder modules from Motion and Context Compression (See Fig. \ref{['fig:framework']}). We first perform downsampling and horizontally shifting operation for the intermediate feature $\bm{K}^L$ (resp., $\bm{K}^R$) to obtain the 4D disparity volume $\bm{V}^L$ (resp., $\bm{V}^R$), where we shift the $\bm{K}^L$ (resp., $\bm{K}^R$) across a disparity range from $1$ to $D$ and $D$ denotes the maximum disparity value. Subsequently, an attention score $\bm{F}^*$ (i.e., similarity map) is generated by applying standard operations, element-wise dot product, $\operatorname{Softplus}$ function and $\operatorname{Tanh}$ function. Then, we apply this score into the previously produced cost-volume $\bm{V}^L$ (resp., $\bm{V}^R$). After a refinement, we can obtain final aligned reference feature $\bm{K}_{Ref}^R$ (resp., $\bm{K}_{Ref}^L$), which will be added back to the intermediate feature $\bm{K}^L$ (resp., $\bm{K}^R$) to produce new intermediate feature $\bm{K}_{new}^L$ (resp., $\bm{K}_{new}^L$).
• Figure 3: (a). Overview of the Hybrid Disparity Compensation module for Entropy Modeling (HDC-EM). The input is the latent feature pair $\{\bm{y}^L, \bm{y}^R\}$ produced by the encoder modules of the Motion and Context Compression components (see the middle part of Fig. \ref{['fig:framework']}). Each latent feature $\bm{y}^L$ (resp., $\bm{y}^R$) is evenly divided along the channel dimension into N slices $\bm{y}_{1\sim N}^L$ (resp., $\bm{y}_{1\sim N}^R$). Meanwhile, spatial–temporal context features $\bm{\varPhi}_{1\sim N}^L$ (resp., $\bm{\varPhi}_{1\sim N}^R$) are generated using the network in (c). To estimate the distribution of each quantized latent slice $\bm{\hat{y}}^L$ (resp., $\bm{\hat{y}}^R$), we use both the context features and the accumulated losslessly compressed slices from the same and opposite views as priors. These are passed to the entropy estimation module $Est_n^M$ in (b) for accurate entropy coding. (b). The entropy estimation network $Est_n^M$ predicts the probability distribution for the n-th slice $\bm{\hat{y}}_n^M$ at the current view $M$, using the accumulated losslessly compressed slices from the same view $M$, (i.e., $\bm{\hat{y}}_{1\sim n-1}^M$) and from the opposite view $\overline{M}$ (i.e., $\bm{\hat{y}}_{1\sim q}^{\overline{M}}$, s.t., $q=n-1$ for $M=L$ and $q=n$ for $M=R$), along with the accumulated context features $\bm{\varPhi}_{1\sim n}^M$. To align the cross-view slices $\bm{\hat{y}}_{1\sim q}^{\overline{M}}$ with the current view, we apply the HDC alignment strategy, which shares a similar architecture to that shown on the right side of Fig. \ref{['fig:framework']}. (c) The spatial–temporal prior fusion network generates the context $\bm{\Phi}^M$ by fusing the hyperprior, extracted from $\bm{y}^M$ via the hyper encoder-decoder ($h_a$ and $h_s$), with the multi-scale features $\bm{Ctx}^M_{1\sim3}$ (see the left part of Fig. \ref{['fig:framework']}). The resulting context is then divided into N channel-wise slices, denoted as $\bm{\Phi}_{1\sim N}^M$, following the same slicing operation used for latent features. The context fusion module adopts a structure similar to the temporal context encoder in DCVC-TCM sheng2022tcm, while $h_a$, ${h_s}$, and the prior fusion module consist of several convolutional layers.
• Figure 4: Rate-distortion (RD) curves. The results are evaluated on the KITTI 2012, KITTI 2015 and Nagoya Nagoya_university_sequences datasets in terms of Bpp-PSNR.
• Figure 5: Subjective quality comparison on KITTI 2012 dataset.