Fast Image-based Neural Relighting with Translucency-Reflection Modeling

TL;DR

This paper tackles sub-second image-based relighting for translucent materials by marrying volumetric neural rendering with precomputed radiance transfer concepts. The proposed TRHM framework decomposes lighting into low-frequency queries via a hypernetwork and high-frequency reflections via a learned local normal, enabling faithful subsurface scattering and glossy reflections in a single pass. The approach is trained in three stages on OLAT data and transferred to envmap lighting using reflection hints and a hypernetwork for the color branch, finally distilling to a fast Hash-Volume Grid. The results demonstrate strong visual fidelity, real-time-like rendering speeds on common GPUs, and high practicality for OLAT-to-envmap relighting with publicly released code and data.

Abstract

Image-based lighting (IBL) is a widely used technique that renders objects using a high dynamic range image or environment map. However, aggregating the irradiance at the object's surface is computationally expensive, in particular for non-opaque, translucent materials that require volumetric rendering techniques. In this paper we present a fast neural 3D reconstruction and relighting model that extends volumetric implicit models such as neural radiance fields to be relightable using IBL. It is general enough to handle materials that exhibit complex light transport effects, such as translucency and glossy reflections from detailed surface geometry, producing realistic and compelling results. Rendering can be within a second at 800$\times$800 resolution (0.72s on an NVIDIA 3090 GPU and 0.30s on an A100 GPU) without engineering optimization. Our code and dataset are available at https://zhusz.github.io/TRHM-Webpage/.

Figures (13)

• Figure 1: Rendering pipeline of the proposed TRHM model. The testing (rendering) undergoes a low-frequency branch (Sec. \ref{['sec:method:lowfreq']}) with a hyper-net predicting based on a low-resolution envmap; and a high-frequency branch (Sec. \ref{['sec:method:reflections']}) where dedicated reflection normal $\hat{n}"$ (Sec. \ref{['sec:method:normals']}) is estimated for facilitating all the reflective high-frequency effects. Only a single rendering pass is necessary for full, image-based illumination. Our training (optimization) undergoes three stages that can effectively transfer from the OLAT data to the envmap prediction mode.
• Figure 2: Rendering pipeline of the proposed TRHM model. The testing (rendering) undergoes a low-frequency branch (Sec. \ref{['sec:method:lowfreq']}) with a hyper-net predicting based on a low-resolution envmap; and a high-frequency branch (Sec. \ref{['sec:method:reflections']}) where dedicated reflection normal $\hat{n}"$ (Sec. \ref{['sec:method:normals']}) is estimated for facilitating all the reflective high-frequency effects. Only a single rendering pass is necessary for full, image-based illumination. Our training (optimization) undergoes three stages that can effectively transfer from the OLAT data to the envmap prediction mode.
• Figure 3: Separation into low-frequency and high-frequency (top: for point light; bottom: for an envmap) enables the proposed framework to render complex materials with high fidelity. In this example, the jade structure is present in the low frequency rendering, but does not exhibit specular highlights. These are captured well in the high frequency component, leading to a faithful rendering. Without the separation, reconstruction of this material is not possible. Best viewed electronically with zoom-in.
• Figure 4: Our model prevails on capturing local micro-geometry details with appealing accuracy of surface normal prediction (c, $\hat{n}"$). Here we visualize our predicted normal for the reflection purpose and compare with the noisy analytical gradient normal (a, $\hat{n}$) as well as the overly-smoothed RefNeRF-predicted normal (b, $\hat{n}'$) (originally proposed for Integrated Direction Encoding verbin2022ref rather than reflections).
• Figure 5: Reflection hint pyramid visualization ($\hat{H}_\text{ref}^\text{LoD}$). The reflection hints model reflected irradiance and take self occlusion into account. As the pyramid level increases, roughness increases and the reflection becomes more blurry. Note they are the input to our high frequency branch rather than our outputs of roughness progression (Fig. 1). The mini irradiance map on the top-left corner of each visualized level (0, 3, 5) shows the filtered input envmap by the gaussian filter.