AIM 2024 Challenge on Efficient Video Super-Resolution for AV1 Compressed Content

TL;DR

The paper addresses the challenge of delivering high-quality video super-resolution for AV1-compressed content in real time, with a focus on mobile and streaming use cases. It introduces a competition-driven framework featuring two tracks (540p→4K and 360p→1080p) and a 4K AV1-encoded test set, evaluated primarily with VMAF alongside PSNR and SSIM under a MAC/compute budget. A suite of lightweight, end-to-end VSR methods is proposed or summarized, including SuperBicubic++, FSMD, BVI-RTVSR, ETDS v2-based approaches, VPEG-VSR, and learnable guided-filter variants, all designed to run efficiently with forward-frame processing and MACs under 250 per frame. The results demonstrate meaningful VMAF improvements over Lanczos baselines while maintaining real-time performance (often under a few milliseconds per frame on modern GPUs) and low parameter counts, underscoring the practicality of real-time VSR in AV1 pipelines. Collectively, these contributions advance practical VSR solutions for high-resolution, compressed video, and provide a benchmark and methodological blueprint for future research and industrial deployment.

Abstract

Video super-resolution (VSR) is a critical task for enhancing low-bitrate and low-resolution videos, particularly in streaming applications. While numerous solutions have been developed, they often suffer from high computational demands, resulting in low frame rates (FPS) and poor power efficiency, especially on mobile platforms. In this work, we compile different methods to address these challenges, the solutions are end-to-end real-time video super-resolution frameworks optimized for both high performance and low runtime. We also introduce a new test set of high-quality 4K videos to further validate the approaches. The proposed solutions tackle video up-scaling for two applications: 540p to 4K (x4) as a general case, and 360p to 1080p (x3) more tailored towards mobile devices. In both tracks, the solutions have a reduced number of parameters and operations (MACs), allow high FPS, and improve VMAF and PSNR over interpolation baselines. This report gauges some of the most efficient video super-resolution methods to date.

Figures (11)

• Figure 1: Frame samples from the high-quality test set videos e.g.Netflix "El Fuente". The original videos are 4K resolution YCbCr 4:2:0 format.
• Figure 2: The challenge framework. The high-resolution (HR) videos are downscaled $\downarrow$ (with Lanczos filter) to lower resolution (LR) videos with 3x, 4x scaling ratio. We encode the videos using SVT–AV1 AV1 and different CRF values to produce encoded video bitstreams with different compression levels. We decode the videos using SVT–AV1, and we upscale the (de)compressed LR videos to the original HR resolution using the proposed video super-resolution (VSR) methods. Finally we evaluate the quality of the reconstructed HR videos (*) using well-known perceptual video quality metrics such as VMAF VMAFli2018vmaf.
• Figure 3: SuperBicubic++ X3 solution.
• Figure 4: SuperBicubic++ X4 solution.
• Figure 5: SuperBicubic++ RepBlock.