$\mathtt{M^3VIR}$: A Large-Scale Multi-Modality Multi-View Synthesized Benchmark Dataset for Image Restoration and Content Creation

TL;DR

The paper presents $\mathcal{M^3VIR}$, a large-scale, multi-modality, multi-view dataset tailored for gaming content restoration and controllable content generation, rendered with Unreal Engine 5 and providing ground-truth LR-HR pairs and synchronized multi-view frames across 80 scenes in 8 categories. It defines two main subsets, $\mathcal{M^3VIR\_MR}$ for media-delivery tasks (SR, NVS, and NVS+SR) and $\mathcal{M^3VIR\_MS}$ for multi-style, object-level controlled video generation, including per-frame depth, segmentation, and exact camera parameters. The authors establish four challenge tracks with baseline evaluations for SR, NVS, and NVS+SR, revealing performance trends across transformers, diffusion models, NeRF-based methods, and 3DGS, as well as inconsistencies in perceptual metrics on gaming data. By releasing $\mathcal{M^3VIR}$, the work provides a valuable benchmark to advance AI-powered restoration, compression, and controllable content generation in next-generation cloud gaming and entertainment, while highlighting the need for more reliable evaluation protocols.

Abstract

The gaming and entertainment industry is rapidly evolving, driven by immersive experiences and the integration of generative AI (GAI) technologies. Training such models effectively requires large-scale datasets that capture the diversity and context of gaming environments. However, existing datasets are often limited to specific domains or rely on artificial degradations, which do not accurately capture the unique characteristics of gaming content. Moreover, benchmarks for controllable video generation remain absent. To address these limitations, we introduce $\mathtt{M^3VIR}$, a large-scale, multi-modal, multi-view dataset specifically designed to overcome the shortcomings of current resources. Unlike existing datasets, $\mathtt{M^3VIR}$ provides diverse, high-fidelity gaming content rendered with Unreal Engine 5, offering authentic ground-truth LR-HR paired and multi-view frames across 80 scenes in 8 categories. It includes $\mathtt{M^3VIR\_MR}$ for super-resolution (SR), novel view synthesis (NVS), and combined NVS+SR tasks, and $\mathtt{M^3VIR\_{MS}}$, the first multi-style, object-level ground-truth set enabling research on controlled video generation. Additionally, we benchmark several state-of-the-art SR and NVS methods to establish performance baselines. While no existing approaches directly handle controlled video generation, $\mathtt{M^3VIR}$ provides a benchmark for advancing this area. By releasing the dataset, we aim to facilitate research in AI-powered restoration, compression, and controllable content generation for next-generation cloud gaming and entertainment.

Figures (7)

• Figure 1: True LR vs. Pseudo LR (downsample+noise+blur)
• Figure 2: Example of 8 scene categories in $\mathtt{M^3VIR}$.
• Figure 3: One data sample example of $\mathtt{M^3VIR\_MR}$. All 6 views have multi-resolution videos. All RGB frames have associated segmentation maps, depth maps and camera intrinsic and extrinsic parameters (details of only one frame is shown).
• Figure 4: A data sample of $\mathtt{M^3VIR\_MS}$ where objects in the scene are changed to different styles.
• Figure 5: Qualitative comparison examples of Real-ESRGAN, DAT, and ResShift for Track 1