Objects With Lighting: A Real-World Dataset for Evaluating Reconstruction and Rendering for Object Relighting

TL;DR

This work introduces Objects With Lighting, a real-world dataset and benchmark for evaluating reconstruction, rendering, and relighting of objects under novel illumination. It pairs ground-truth environment maps with posed images and calibrated geometry, enabling quantitative assessment of relighting fidelity using a simple Mitsuba+NeuS baseline and a broad set of state-of-the-art methods. The experiments reveal that relighting under unseen lighting is more challenging than standard novel-view synthesis and expose failure modes not evident in synthetic data. By providing code and evaluation tools, the dataset aims to accelerate progress in real-world inverse rendering and relighting research.

Abstract

Reconstructing an object from photos and placing it virtually in a new environment goes beyond the standard novel view synthesis task as the appearance of the object has to not only adapt to the novel viewpoint but also to the new lighting conditions and yet evaluations of inverse rendering methods rely on novel view synthesis data or simplistic synthetic datasets for quantitative analysis. This work presents a real-world dataset for measuring the reconstruction and rendering of objects for relighting. To this end, we capture the environment lighting and ground truth images of the same objects in multiple environments allowing to reconstruct the objects from images taken in one environment and quantify the quality of the rendered views for the unseen lighting environments. Further, we introduce a simple baseline composed of off-the-shelf methods and test several state-of-the-art methods on the relighting task and show that novel view synthesis is not a reliable proxy to measure performance. Code and dataset are available at https://github.com/isl-org/objects-with-lighting .

Figures (22)

• Figure 1: Overview of the Objects With Lighting dataset. First row: Our dataset contains 8 objects which we capture in 3 different lighting conditions each. Second and third row: We show 2 HDR environment map examples for each lighting category which are (left to right): indoor with natural light, indoor with artificial light, and outdoor.
• Figure 2: We take 42-67 photos for each object and roughly sample a hemisphere with the object in the center.
• Figure 3: Test protocol on the Objects With Lighting dataset. We reconstruct a relightable representation of the object from a set of images taken in one environment. The reconstruction can be a textured mesh, a neural representation, or any other representation that can be relit with a new environment map and rendered from a novel view point. Testing is conducted by rendering the objects using environment maps from the reconstruction environment and new environments and comparing the rendered images to the corresponding test images of that environment.
• Figure 4: Qualitative examples for relighting on our dataset. The first row shows the environment map used for relighting. For each object, we show one of the test images of the 3 lighting categories. The first image for each object corresponds to the reconstruction environment. Each object has been captured in highly different lighting conditions: outdoor, indoor, and indoor with artificial light.
• Figure 5: Visualization of the coordinate systems used with environment maps.