3D Scene Creation and Rendering via Rough Meshes: A Lighting Transfer Avenue

TL;DR

This work addresses rendering with rough 3D models (R3DMs) by bridging neural fields rendering (NFR) and physically-based rendering (PBR) through a Lighting Transfer Network (LighTNet). LighTNet uses a simplified image composition model and a residual shading term to transfer lighting details from PBR shading to NeRF-rendered content, augmented by a Lab Angle loss to improve lighting-color contrast. It is trained on synthetic <R3DM, 3DM> pairs and generalizes to unseen R3DMs and arbitrary lighting, enabling scene creation with freely edited lighting without per-scene optimization. Experiments on the 3DF-Lighting dataset show improved PSNR, Structural Similarity, and a dedicated Lab Angle metric, and qualitative results demonstrate realistic local shadows and lighting interactions. The approach offers a practical pathway for integrating NeRF-based objects into complex 3D workflows for AR/VR, design, and visualization.

Abstract

This paper studies how to flexibly integrate reconstructed 3D models into practical 3D modeling pipelines such as 3D scene creation and rendering. Due to the technical difficulty, one can only obtain rough 3D models (R3DMs) for most real objects using existing 3D reconstruction techniques. As a result, physically-based rendering (PBR) would render low-quality images or videos for scenes that are constructed by R3DMs. One promising solution would be representing real-world objects as Neural Fields such as NeRFs, which are able to generate photo-realistic renderings of an object under desired viewpoints. However, a drawback is that the synthesized views through Neural Fields Rendering (NFR) cannot reflect the simulated lighting details on R3DMs in PBR pipelines, especially when object interactions in the 3D scene creation cause local shadows. To solve this dilemma, we propose a lighting transfer network (LighTNet) to bridge NFR and PBR, such that they can benefit from each other. LighTNet reasons about a simplified image composition model, remedies the uneven surface issue caused by R3DMs, and is empowered by several perceptual-motivated constraints and a new Lab angle loss which enhances the contrast between lighting strength and colors. Comparisons demonstrate that LighTNet is superior in synthesizing impressive lighting, and is promising in pushing NFR further in practical 3D modeling workflows.

Figures (18)

• Figure 1: A Lighting Transfer Avenue.Left (Problem): Given some reconstructed rough 3D models (R3DMs) and designed 3D CAD models (3DMs), artists can use them to create any 3D scenes and freely perform arbitrary lighting simulation for each created scene. A physically-based rendering (PBR) system can only render low-quality images or videos for these scenes. Our goal is to render high-quality content from these possible scenes without training (or fitting) the newly created 3D scenes. Right (Solution): As an example, if we have pre-obtained a neural fields representation (e.g. NeRF mildenhall2020nerf) for each real object, we can synthesize object instances for R3DMs in impressive quality through neural fields rendering (NFR). Unluckily, NFR instances cannot reflect the simulated lighting details (e.g., local shadows) on R3DMs. We propose a lighting transfer network (LighTNet) to bridge NFR and PBR, such that they can benefit from each other. In practice, LighTNet is trained once in a dataset and can be used for all the newly created 3D scenes with both seen and unseen R3DMs and arbitrary lighting (See "Generalizing to Real-Lighting" in Fig. \ref{['fig:real']}).
• Figure 2: Training a LighTNet. LighTNet aims to transfer the lighting details from an imperfect shading map $\mathcal{S}'$ to the corresponding image $\mathcal{I}_s$. It reasons about the reformulated image composition model $\mathcal{I}_t = (\mathcal{D} + \mathcal{R} + \alpha) \cdot (\mathcal{S}' + \mathcal{S}'_r)$. The yellow (left) part shows the $<$R3DM, 3DM$>$ pairs generation process, and is only included in the training process. Once optimized, LighTNet can be used for any newly created 3D scenes with both seen and unseen R3DMs and support free lighting simulation. In the inference phase, $\mathcal{I}_s$ of an object is the 2D instance synthesized by a trained NeRF or any other high-performing free view synthesis formulations (See Fig. \ref{['fig:relighting']} and Fig. \ref{['fig:nvr3d-sd']}).
• Figure 3: Compositing Individual NeRF Objects. Several concurrent works chen2022mobilenerfmetaconnect22 show it's possible to represent real-world objects as individual NeRFs and R3DMs for freely 3D scene creation and rendering. LighTNet goes a futher step by considering the indirect lighting effects such as local shadows on R3DMs caused by objects interactions.
• Figure 4: 3D Scene Creation and Rendering via R3DMs. We can represent real-world objects as individual NeRFs and R3DMs, and freely composite them to create unlimited 3D scenes. After lighting editing by artists, LighTNet can transfer direct and indirect lighting effects on R3DMs (e.g. $\mathcal{S}'$) to the corresponding NFR instances (e.g. $\mathcal{I}_s$). See Sec. \ref{['subsec:nvr3d-lightnet']} for the detailed explanation.
• Figure 5: Training Set Construction. We take the "Sofa" case as an example to show how to capture a training sample $\{ (\mathcal{I}_s, \mathcal{S}', \bar{\mathcal{I}}_{t}, \bar{\mathcal{D}}, \bar{\mathcal{R}}\}$ via 3D CAD models and 3D scenes. The elements are rendered by Blenderblender. LighTNet is trained once on the 3DF-Lighting training set, and can be used for all the newly created scenes with both seen and unseen R3DMs and arbitrary lighting.