Sliding Window Attention for Learned Video Compression

TL;DR

The paper tackles inefficiencies of patch-based transformers in learned video compression by introducing 3D Sliding Window Attention, a patchless, decoder-only autoregressive entropy model that unifies spatial and temporal context with a uniform receptive field. It demonstrates BD-rate savings up to about 18.6% over the baseline and significant reductions in decoder and entropy-model complexity. The approach yields robust gains across datasets, while ablations reveal an optimal temporal context length and a trade-off between long-range information and relevance. The work lays a foundation for future improvements, including parallel decoding and hyperprior integration to surpass state-of-the-art codecs.

Abstract

To manage the complexity of transformers in video compression, local attention mechanisms are a practical necessity. The common approach of partitioning frames into patches, however, creates architectural flaws like irregular receptive fields. When adapted for temporal autoregressive models, this paradigm, exemplified by the Video Compression Transformer (VCT), also necessitates computationally redundant overlapping windows. This work introduces 3D Sliding Window Attention (SWA), a patchless form of local attention. By enabling a decoder-only architecture that unifies spatial and temporal context processing, and by providing a uniform receptive field, our method significantly improves rate-distortion performance, achieving Bjørntegaard Delta-rate savings of up to 18.6 % against the VCT baseline. Simultaneously, by eliminating the need for overlapping windows, our method reduces overall decoder complexity by a factor of 2.8, while its entropy model is nearly 3.5 times more efficient. We further analyze our model's behavior and show that while it benefits from long-range temporal context, excessive context can degrade performance.

Figures (4)

• Figure 1: Visualization of the proposed method. (a) The 3D kernel. For the current hyperpixel (purple), attention is computed over its previously decoded spatio-temporal neighbors (blue). Future hyperpixels that fall within the window (transparent) are causally masked. (b) The proposed transformer block architecture.
• Figure 1: -rate savings (%) relative to the baseline for all test datasets.
• Figure 2: Autoregressive prediction scheme for a single frame. Decoding follows a line-scan order (green). At the start of each new row, the hyperpixel directly above it is prepended to the input sequence (red), ensuring the local attention window has access to spatially relevant context.
• Figure 4: Relative -rate as a function of the number of reference frames for the HEVC B and UVG datasets. The optimal context size for each sequence is used as the 0% reference.