Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SR+Codec: a Benchmark of Super-Resolution for Video Compression Bitrate Reduction

Evgeney Bogatyrev, Ivan Molodetskikh, Dmitriy Vatolin

TL;DR

The paper addresses how super-resolution (SR) can be leveraged to reduce video compression bitrate without sacrificing perceptual quality. It introduces a benchmark evaluating 19 SR models across five codecs, using both objective metrics and crowdsourced subjective assessments to measure bitrate reduction and restoration quality. Key findings show codec-dependent SR effectiveness, with significant bitrate savings (up to 65%) and improved alignment between subjective quality and a combined ERQA×MDTVSFA metric, though standard metrics often poorly correlate with human judgments. The work provides a publicly available benchmark and practical guidance for selecting SR methods in compression scenarios, with implications for streaming efficiency and future SR-codec co-design.

Abstract

In recent years, there has been significant interest in Super-Resolution (SR), which focuses on generating a high-resolution image from a low-resolution input. Deep learning-based methods for super-resolution have been particularly popular and have shown impressive results on various benchmarks. However, research indicates that these methods may not perform as well on strongly compressed videos. We developed a super-resolution benchmark to analyze SR's capacity to upscale compressed videos. Our dataset employed video codecs based on five widely-used compression standards: H.264, H.265, H.266, AV1, and AVS3. We assessed 19 popular SR models using our benchmark and evaluated their ability to restore details and their susceptibility to compression artifacts. To get an accurate perceptual ranking of SR models, we conducted a crowd-sourced side-by-side comparison of their outputs. We found that some SR models, combined with compression, allow us to reduce the video bitrate without significant loss of quality. We also compared a range of image and video quality metrics with subjective scores to evaluate their accuracy on super-resolved compressed videos. The benchmark is publicly available at https://videoprocessing.ai/benchmarks/super-resolution-for-video-compression.html

SR+Codec: a Benchmark of Super-Resolution for Video Compression Bitrate Reduction

TL;DR

The paper addresses how super-resolution (SR) can be leveraged to reduce video compression bitrate without sacrificing perceptual quality. It introduces a benchmark evaluating 19 SR models across five codecs, using both objective metrics and crowdsourced subjective assessments to measure bitrate reduction and restoration quality. Key findings show codec-dependent SR effectiveness, with significant bitrate savings (up to 65%) and improved alignment between subjective quality and a combined ERQA×MDTVSFA metric, though standard metrics often poorly correlate with human judgments. The work provides a publicly available benchmark and practical guidance for selecting SR methods in compression scenarios, with implications for streaming efficiency and future SR-codec co-design.

Abstract

In recent years, there has been significant interest in Super-Resolution (SR), which focuses on generating a high-resolution image from a low-resolution input. Deep learning-based methods for super-resolution have been particularly popular and have shown impressive results on various benchmarks. However, research indicates that these methods may not perform as well on strongly compressed videos. We developed a super-resolution benchmark to analyze SR's capacity to upscale compressed videos. Our dataset employed video codecs based on five widely-used compression standards: H.264, H.265, H.266, AV1, and AVS3. We assessed 19 popular SR models using our benchmark and evaluated their ability to restore details and their susceptibility to compression artifacts. To get an accurate perceptual ranking of SR models, we conducted a crowd-sourced side-by-side comparison of their outputs. We found that some SR models, combined with compression, allow us to reduce the video bitrate without significant loss of quality. We also compared a range of image and video quality metrics with subjective scores to evaluate their accuracy on super-resolved compressed videos. The benchmark is publicly available at https://videoprocessing.ai/benchmarks/super-resolution-for-video-compression.html
Paper Structure (14 sections, 5 figures, 4 tables)

This paper contains 14 sections, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Super-resolution methods
  4. Downscale-based video codecs
  5. Super-resolution benchmarks
  6. Benchmark Methodology
  7. Dataset preparation
  8. Benchmark pipeline
  9. Quality estimation and subjective study
  10. Evaluation Results
  11. Comparison by resulting video quality
  12. Comparison by bitrate reduction
  13. Video quality metrics assessment
  14. Conclusion

Figures (5)

  • Figure 1: The comparison between 3 super-resolution models on compressed video sequence. COMISR eliminates the compression artifact because it is designed to work with compressed video. On the other hand, VRT and Real-ESRGAN fail to remove this artifact.
  • Figure 2: Distribution of Google Spatial and Temporal wang2019youtube information for videos we considered when creating our training dataset. Chosen videos appear in orange, others in blue.
  • Figure 3: Example videos from the dataset. The dataset includes real-world sequences, animation, and clips from games.
  • Figure 4: The evaluation pipeline of our benchmark. The pipeline consists of three steps: $4\times$ bicubic downscaling, compression, and $4\times$ SR upscaling.
  • Figure 5: RealSR applied to video compressed with x264 codec at 1.2 Mbps can achieve the same visual quality as plain x264 codec at 3.8 Mbps.