Neural LightRig: Unlocking Accurate Object Normal and Material Estimation with Multi-Light Diffusion

TL;DR

Neural LightRig addresses the under-constrained problem of recovering object geometry and material from a single image by leveraging diffusion-based multi-light priors to enrich lighting information. It couples a multi-light diffusion model with a large G-buffer regression network to predict normals and PBR maps, trained on a synthetic LightProp dataset designed for consistent lighting variation and domain alignment. The approach achieves state-of-the-art results in normal and material estimation and produces realistic relighting across diverse lighting, with code and data released for reproducibility. This has practical impact for realistic rendering, AR/VR, and robotics by enabling accurate monocular intrinsic estimation under varied illumination.

Abstract

Recovering the geometry and materials of objects from a single image is challenging due to its under-constrained nature. In this paper, we present Neural LightRig, a novel framework that boosts intrinsic estimation by leveraging auxiliary multi-lighting conditions from 2D diffusion priors. Specifically, 1) we first leverage illumination priors from large-scale diffusion models to build our multi-light diffusion model on a synthetic relighting dataset with dedicated designs. This diffusion model generates multiple consistent images, each illuminated by point light sources in different directions. 2) By using these varied lighting images to reduce estimation uncertainty, we train a large G-buffer model with a U-Net backbone to accurately predict surface normals and materials. Extensive experiments validate that our approach significantly outperforms state-of-the-art methods, enabling accurate surface normal and PBR material estimation with vivid relighting effects. Code and dataset are available on our project page at https://projects.zxhezexin.com/neural-lightrig.

Figures (19)

• Figure 1: Neural LightRig takes an image as input and generates multi-light images to assist the estimation of high-quality normal and PBR materials, which can be used to render realistic relit images under various environment lighting.
• Figure 2: Framework Overview. Multi-light diffusion generates multi-light images from an input image. These images with corresponding lighting orientations are then used to predict surface normals and PBR materials with a regression U-Net.
• Figure 3: Hybrid condition in multi-light diffusion. Input images are incorporated via concatenation with noise latents and enhanced through reference attention, where queries in the denoise stream attend to keys and values from both streams.
• Figure 4: Visualization of multi-light setup in LightProp. Camera and point lights are positioned on a sphere around the object. $\theta, \varphi$ are spherical coordinates to determine each light's orientation relative to the object.
• Figure 5: Qualitative comparison on surface normal estimation. Ground truth normals (G.T.) are provided for input images rendered from available 3D objects (the last two rows) and are omitted for in-the-wild images (the first two rows).