GameIR: A Large-Scale Synthesized Ground-Truth Dataset for Image Restoration over Gaming Content

TL;DR

This work develops GameIR, a large-scale high-quality computer-synthesized ground-truth dataset to fill in the blanks over gaming content, and evaluates several SOTA super-resolution algorithms and NeRF-based NVS algorithms over the dataset, which demonstrates the effectiveness of the ground-truth GameIR data in improving restoration performance for gaming content.

Abstract

Image restoration methods like super-resolution and image synthesis have been successfully used in commercial cloud gaming products like NVIDIA's DLSS. However, restoration over gaming content is not well studied by the general public. The discrepancy is mainly caused by the lack of ground-truth gaming training data that match the test cases. Due to the unique characteristics of gaming content, the common approach of generating pseudo training data by degrading the original HR images results in inferior restoration performance. In this work, we develop GameIR, a large-scale high-quality computer-synthesized ground-truth dataset to fill in the blanks, targeting at two different applications. The first is super-resolution with deferred rendering, to support the gaming solution of rendering and transferring LR images only and restoring HR images on the client side. We provide 19200 LR-HR paired ground-truth frames coming from 640 videos rendered at 720p and 1440p for this task. The second is novel view synthesis (NVS), to support the multiview gaming solution of rendering and transferring part of the multiview frames and generating the remaining frames on the client side. This task has 57,600 HR frames from 960 videos of 160 scenes with 6 camera views. In addition to the RGB frames, the GBuffers during the deferred rendering stage are also provided, which can be used to help restoration. Furthermore, we evaluate several SOTA super-resolution algorithms and NeRF-based NVS algorithms over our dataset, which demonstrates the effectiveness of our ground-truth GameIR data in improving restoration performance for gaming content. Also, we test the method of incorporating the GBuffers as additional input information for helping super-resolution and NVS. We release our dataset and models to the general public to facilitate research on restoration methods over gaming content.

Figures (6)

• Figure 1: Representative views of 8 different towns.
• Figure 2: An example of GameIR-SR dataset, consisting of paired LR-HR (LR top row, HR bottom row) RGB images with associated GBuffer information.
• Figure 3: True LR input versus pseudo LR input. The pseudo LR input is downsampled from the HR input with added noise and blur as commonly used degradationzhang2017learningzhang2018residual.
• Figure 4: An example of our GameIR-NVS dataset, consisting of 6 views of RGB images with associated segmentation maps and depth maps. From the top to bottom, the sequence gives views of a 360° rotation, starting from the front.
• Figure 5: Qualitative comparison of pretrained (PT) and finetuned (FT) super-resolution methods. Finetuning with ground-truth LR-HR paired data can restore intricate details more accurately with better clarity.