OpenMaterial: A Large-scale Dataset of Complex Materials for 3D Reconstruction

TL;DR

OpenMaterial addresses the challenge of material-aware 3D reconstruction by providing a large-scale semi-synthetic dataset that uses spectral-valued IORs to simulate wavelength-dependent light transport within a physically-based rendering framework. The dataset includes 1001 shapes, 295 materials, 714 HDRIs, and full annotations (images, meshes, poses, depth, masks) to benchmark 3D reconstruction and novel-view synthesis, modeled through the BSDF decomposition $f_s=f_r+f_t$ with Fresnel $F$, geometry term $G$, and microfacet distribution $D$. The authors benchmark 11 SOTA methods, perform ablations across material, geometry, and illumination, and demonstrate significant performance gaps on non-diffuse materials, underscoring the need for material-aware, physics-informed approaches with potential impact on robotics, digital twins, and synthetic data pipelines. This work provides a strong, reusable platform for advancing robust 3D reconstruction under real-world optical complexities and enables targeted development of methods that better handle specularities, refractions, and transparency.

Abstract

Recent advances in deep learning, such as neural radiance fields and implicit neural representations, have significantly advanced 3D reconstruction. However, accurately reconstructing objects with complex optical properties, such as metals, glass, and plastics, remains challenging due to the breakdown of multi-view color consistency in the presence of specular reflections, refractions, and transparency. This limitation is further exacerbated by the lack of benchmark datasets that explicitly model material-dependent light transport. To address this, we introduce OpenMaterial, a large-scale semi-synthetic dataset for benchmarking material-aware 3D reconstruction. It comprises 1,001 objects spanning 295 distinct materials, including conductors, dielectrics, plastics, and their roughened variants, captured under 714 diverse lighting conditions. By integrating lab-measured Index of Refraction (IOR) spectra, OpenMaterial enables the generation of high-fidelity multi-view images that accurately simulate complex light-matter interactions. It provides multi-view images, 3D shape models, camera poses, depth maps, and object masks, establishing the first extensive benchmark for evaluating 3D reconstruction on challenging materials. We evaluate 11 state-of-the-art methods for 3D reconstruction and novel view synthesis, conducting ablation studies to assess the impact of material type, shape complexity, and illumination on reconstruction performance. Our results indicate that OpenMaterial provides a strong and fair basis for developing more robust, physically-informed 3D reconstruction techniques to better handle real-world optical complexities.

Figures (5)

• Figure 1: Comparison of Material Properties under Uniform Lighting Conditions. We illustrate the distinct visual appearances of seven objects made from different materials—conductor, dielectric, plastic, their rough versions, and diffuse—rendered under the same lighting and scene settings to highlight how material composition influences appearance. Accompanying each object are diagrams of the physical principles that explain the specific ways in which light interacts with these materials, emphasizing the substantial impact of material characteristics on visual perception.
• Figure 2: Illustration of selected HDRI environment maps.
• Figure 3: Selection of objects.
• Figure 4: Qualitative 3D Reconstruction Results. Top: Input images; 2nd row: Ground truth; 3rd row: PGSR; Bottom: NeRO.
• Figure 5: Qualitative results for novel view synthesis. From left to right: GT (global), GT (local), PGSR, Instant-NGP. Additional results are provided in the supplementary material.