Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AIM 2024 Sparse Neural Rendering Challenge: Dataset and Benchmark

Michal Nazarczuk, Thomas Tanay, Sibi Catley-Chandar, Richard Shaw, Radu Timofte, Eduardo Pérez-Pellitero

TL;DR

This work introduces a new dataset that follows the setup of the DTU MVS dataset, and proposes two different sparse configurations, which provide a powerful and convenient tool for reproducible evaluation, and enable researchers easy access to a public leaderboard with the state-of-the-art performance scores.

Abstract

Recent developments in differentiable and neural rendering have made impressive breakthroughs in a variety of 2D and 3D tasks, e.g. novel view synthesis, 3D reconstruction. Typically, differentiable rendering relies on a dense viewpoint coverage of the scene, such that the geometry can be disambiguated from appearance observations alone. Several challenges arise when only a few input views are available, often referred to as sparse or few-shot neural rendering. As this is an underconstrained problem, most existing approaches introduce the use of regularisation, together with a diversity of learnt and hand-crafted priors. A recurring problem in sparse rendering literature is the lack of an homogeneous, up-to-date, dataset and evaluation protocol. While high-resolution datasets are standard in dense reconstruction literature, sparse rendering methods often evaluate with low-resolution images. Additionally, data splits are inconsistent across different manuscripts, and testing ground-truth images are often publicly available, which may lead to over-fitting. In this work, we propose the Sparse Rendering (SpaRe) dataset and benchmark. We introduce a new dataset that follows the setup of the DTU MVS dataset. The dataset is composed of 97 new scenes based on synthetic, high-quality assets. Each scene has up to 64 camera views and 7 lighting configurations, rendered at 1600x1200 resolution. We release a training split of 82 scenes to foster generalizable approaches, and provide an online evaluation platform for the validation and test sets, whose ground-truth images remain hidden. We propose two different sparse configurations (3 and 9 input images respectively). This provides a powerful and convenient tool for reproducible evaluation, and enable researchers easy access to a public leaderboard with the state-of-the-art performance scores. Available at: https://sparebenchmark.github.io/

AIM 2024 Sparse Neural Rendering Challenge: Dataset and Benchmark

TL;DR

This work introduces a new dataset that follows the setup of the DTU MVS dataset, and proposes two different sparse configurations, which provide a powerful and convenient tool for reproducible evaluation, and enable researchers easy access to a public leaderboard with the state-of-the-art performance scores.

Abstract

Recent developments in differentiable and neural rendering have made impressive breakthroughs in a variety of 2D and 3D tasks, e.g. novel view synthesis, 3D reconstruction. Typically, differentiable rendering relies on a dense viewpoint coverage of the scene, such that the geometry can be disambiguated from appearance observations alone. Several challenges arise when only a few input views are available, often referred to as sparse or few-shot neural rendering. As this is an underconstrained problem, most existing approaches introduce the use of regularisation, together with a diversity of learnt and hand-crafted priors. A recurring problem in sparse rendering literature is the lack of an homogeneous, up-to-date, dataset and evaluation protocol. While high-resolution datasets are standard in dense reconstruction literature, sparse rendering methods often evaluate with low-resolution images. Additionally, data splits are inconsistent across different manuscripts, and testing ground-truth images are often publicly available, which may lead to over-fitting. In this work, we propose the Sparse Rendering (SpaRe) dataset and benchmark. We introduce a new dataset that follows the setup of the DTU MVS dataset. The dataset is composed of 97 new scenes based on synthetic, high-quality assets. Each scene has up to 64 camera views and 7 lighting configurations, rendered at 1600x1200 resolution. We release a training split of 82 scenes to foster generalizable approaches, and provide an online evaluation platform for the validation and test sets, whose ground-truth images remain hidden. We propose two different sparse configurations (3 and 9 input images respectively). This provides a powerful and convenient tool for reproducible evaluation, and enable researchers easy access to a public leaderboard with the state-of-the-art performance scores. Available at: https://sparebenchmark.github.io/
Paper Structure (20 sections, 9 figures, 4 tables)

This paper contains 20 sections, 9 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Datasets
  4. Sparse View Rendering
  5. Current Protocol Overview
  6. Dataset
  7. Scene Composition
  8. Generation
  9. Camera Positioning
  10. Lighting Setup
  11. Evaluation Protocol
  12. Data Splits
  13. Input Views
  14. Sparse Neural Rendering Challenge
  15. Benchmark Platform
  16. ...and 5 more sections

Figures (9)

  • Figure 1: Three versions of the same image extracted from different dataset processing pipelines.
  • Figure 2: Composition of high-level categories of objects in SpaRe.
  • Figure 3: Examples of objects included in the SpaRe dataset. We provide objects from diverse categories with a scene placement similar to that of the DTU dataset.
  • Figure 4: Varying illumination in the SpaRe dataset capture for three different scenes.
  • Figure 5: A diagram marking the selection of input views among $49$ available cameras. Input views are presented for the following scenes left to right: Rings, Recorder, Jenga. Selected input views are marked in blue.
  • ...and 4 more figures