RefRef: A Synthetic Dataset and Benchmark for Reconstructing Refractive and Reflective Objects

TL;DR

RefRef tackles 3D reconstruction of scenes with refractive and reflective objects by introducing a synthetic dataset of 50 objects across geometry/material categories, rendered in three backgrounds to yield 150 scenes. It pairs this with an oracle that uses ground-truth geometry and refractive indices to compute accurate light paths and a practical relaxation, R3F, that estimates geometry to approach the oracle. Benchmarking against leading NeRF-like methods reveals substantial performance gaps, especially in complex light transport scenarios involving multiple refractions and total internal reflection. The work underscores the need for new reconstruction techniques that reliably account for nonlinear light paths in transparent and reflective materials, with potential implications for robotics, AR/VR, and outdoor perception.

Abstract

Modern 3D reconstruction and novel view synthesis approaches have demonstrated strong performance on scenes with opaque Lambertian objects. However, most assume straight light paths and therefore cannot properly handle refractive and reflective materials. Moreover, datasets specialized for these effects are limited, stymieing efforts to evaluate performance and develop suitable techniques. In this work, we introduce a synthetic RefRef dataset and benchmark for reconstructing scenes with refractive and reflective objects from posed images. Our dataset has 50 such objects of varying complexity, from single-material convex shapes to multi-material non-convex shapes, each placed in three different background types, resulting in 150 scenes. We also propose an oracle method that, given the object geometry and refractive indices, calculates accurate light paths for neural rendering, and an approach based on this that avoids these assumptions. We benchmark these against several state-of-the-art methods and show that all methods lag significantly behind the oracle, highlighting the challenges of the task and dataset.

Figures (7)

• Figure 1: Comparison of light interactions with opaque Lambertian objects and refractive/reflective objects. (\ref{['fig:splash_a']}) Opaque Lambertian objects allow modeling of light paths as linear. (\ref{['fig:splash_b']}) Refractive and reflective objects lead to the curving and branching of light paths. (\ref{['fig:splash_c']}) Novel view synthesis results. The Oracle performs best, while R3F loses some high-frequency detail and TNSR Deng:tnsr sacrifices geometric quality for visual crispness. (\ref{['fig:splash_d']}) Sample scenes from the RefRef dataset.
• Figure 2: Overview of the oracle method. The process begins with the generation of a straight ray (blue arrow) and sample points along its path. The scene's geometry and refractive index (IoR) are then used to update the ray trajectory, as detailed in the red dashed box on the right. Here, refracted and reflected rays are handled separately, resulting in two sets of updated sample point positions and directions, which are subsequently processed by the Zip-NeRF field that predicts the color ${\mathbf{c}}_i$ and density $\sigma_i$ for each sample point. Using \ref{['eq:rendering']}, the final color along each ray is rendered and the refraction and reflection paths are combined via \ref{['eq:fresnel']}. Note that Zip-NeRF samples points in a conical spiral; we only visualize the centerline of the cone for clarity.
• Figure 3: Comparison of the original barron2022mip distortion loss ${\mathcal{L}}_\text{dist}^\text{orig}$ and the translucency-corrected distortion loss ${\mathcal{L}}_{\text{dist}}$. (\ref{['fig:distortion_a']})--(\ref{['fig:distortion_b']}) Distribution of sample points. (\ref{['fig:distortion_c']})--(\ref{['fig:distortion_d']}) Distribution of weights.
• Figure 4: Qualitative comparison of novel view synthesis and distance maps across four scenes using Zip-NeRF barron2023zipnerf, MS-NeRF msnerf, Ray Deformation Network raydeform, TNSR Deng:tnsr, R3F (Ours), and Oracle. R3F and Oracle render more accurate results, especially in scenes with multiple refractions and total internal reflection, where other methods often fail.
• Figure 5: Failure case of the oracle and R3F methods. The oracle method, despite access to ground-truth geometry and refractive indices, struggles to model the vase’s uneven surface. Meanwhile, R3F treats the vase as solid, causing appearance distortions near the top.