Deep Joint Source-Channel Coding for Wireless Video Transmission with Asymmetric Context

TL;DR

This work tackles wireless video transmission by addressing the cliff effect and suboptimality of separated source-channel coding. It introduces a deep JSCC framework that learns encoding/decoding conditions from asymmetric contexts, augments it with feature propagation to exploit long-range temporal correlations, and enables variable bandwidth through a mask-based entropy module guided by an entropy model. The approach differentiates I-frame and P-frame coding, employs a motion-aware conditioning mechanism, and uses content-adaptive masking to allocate bandwidth efficiently, achieving superior rate-distortion performance over state-of-the-art DeepJSCC and traditional digital schemes. The results demonstrate robust performance under varying channel conditions and GOP lengths, with notable gains in PSNR, MS-SSIM, and perceptual quality (LPIPS).

Abstract

In this paper, we propose a high-efficiency deep joint source-channel coding (JSCC) method for video transmission based on conditional coding with asymmetric context. The conditional coding-based neural video compression requires to predict the encoding and decoding conditions from the same context which includes the same reconstructed frames. However in JSCC schemes which fall into pseudo-analog transmission, the encoder cannot infer the same reconstructed frames as the decoder even a pipeline of the simulated transmission is constructed at the encoder. In the proposed method, without such a pipeline, we guide and design neural networks to learn encoding and decoding conditions from asymmetric contexts. Additionally, we introduce feature propagation, which allows intermediate features to be independently propagated at the encoder and decoder and help to generate conditions, enabling the framework to greatly leverage temporal correlation while mitigating the problem of error accumulation. To further exploit the performance of the proposed transmission framework, we implement content-adaptive coding which achieves variable bandwidth transmission using entropy models and masking mechanisms. Experimental results demonstrate that our method outperforms existing deep video transmission frameworks in terms of performance and effectively mitigates the error accumulation. By mitigating the error accumulation, our schemes can reduce the frequency of inserting intra-frame coding modes, further enhancing performance.

Figures (19)

• Figure 1: Comparison for the relevant methods.
• Figure 2: Overall framework of the Proposed Schemes. Given an input frame $x_t$, the motion vector $v_t$ between $x_t$ and previous frame $x_{t-1}$ is estimated by $ME$. The condition generation network $Cond$ generates asymmetric contexts $c_t$ and $\hat{c}_t$ at the encoder and decoder, respectively. These contexts are subsequently fed into the feature encoder/decoder $F_{Pe}/F_{Pd}$ to perform the feature encoding and reconstruction of the current frame $x_t$. The gray dashed arrow indicates the propagation of the original and reconstruted frames. The black solid arrow represents the data flow at the encoder and decoder through the wireless channel. The yellow flow represents the propagation of the feature.
• Figure 3: Network architectures of I-frame encoder and decoder. The feature encoder $F_{Ie}$ extracts spatial features from the input frame, while the feature decoder $F_{Id}$ and $Refine$ reconstructs the frame at the decoder through feature decoding and refinement. $(M, N)$ represents the kernel size and number of output channels, $s2$ denotes stride of 2 and the followed $\uparrow/\downarrow$ indicates upsampling or downsampling.
• Figure 4: Network architectures of the MV encoder and decoder. The MV encoder $MV_e$ and decoder $MV_d$ are responsible for encoding and decoding motion vectors respectively.
• Figure 5: Network architectures of the P-frame encoder and decoder. The asymmetric contexts are generated by the condition generation network $Cond$ at the encoder and decoder, and then fed into the corresponding feature encoder $F_{Pe}$ and decoder $F_{Pd}$.