Rotated Lights for Consistent and Efficient 2D Gaussians Inverse Rendering

TL;DR

Inverse rendering with 2D Gaussian splatting faces color-illumination ambiguity in albedo estimation. The authors introduce RotLight, a rotated-light capture setup, along with a proxy mesh for incident queries and a residual constraint for the radiance cache to better model global illumination. They demonstrate improved albedo fidelity and efficient computation on synthetic and real-world data, with ablations confirming the contributions. The approach provides a practical, scalable path toward robust inverse rendering under rotated lighting and offers data/code for reproducibility.

Abstract

Inverse rendering aims to decompose a scene into its geometry, material properties and light conditions under a certain rendering model. It has wide applications like view synthesis, relighting, and scene editing. In recent years, inverse rendering methods have been inspired by view synthesis approaches like neural radiance fields and Gaussian splatting, which are capable of efficiently decomposing a scene into its geometry and radiance. They then further estimate the material and lighting that lead to the observed scene radiance. However, the latter step is highly ambiguous and prior works suffer from inaccurate color and baked shadows in their albedo estimation albeit their regularization. To this end, we propose RotLight, a simple capturing setup, to address the ambiguity. Compared to a usual capture, RotLight only requires the object to be rotated several times during the process. We show that as few as two rotations is effective in reducing artifacts. To further improve 2DGS-based inverse rendering, we additionally introduce a proxy mesh that not only allows accurate incident light tracing, but also enables a residual constraint and improves global illumination handling. We demonstrate with both synthetic and real world datasets that our method achieves superior albedo estimation while keeping efficient computation.

Figures (13)

• Figure 1: Overview of our pipeline. RotLight: we capture the same object with $K=3$ different rotations. Stage 1: we pretrain a 2DGS model with a set of images under the same lights (e.g. $\phi_1$) to obtain an initial geometry reconstruction of the scene. Stage 2: we obtain pixel material, depth and surface normal by splatting Gaussians. The depth and normal are used to query incident lights, with direct illumination from the environment map $\textbf{E}_{\phi_k}$ (Section \ref{['sec:method_rotlight']}), and indirect illumination from a radiance cache (Section \ref{['sec:method_radiance_cache']}). The radiance cache is further supervised with a residual constraint evaluated at random surface points. For accurate incident sampling, we also use a proxy mesh instead of ray tracing the Gaussians (Section \ref{['sec:method_incmesh']}).
• Figure 2: Examples from our RotLight dataset. The three rows are light rotations at 0, 120 and 240 degrees respectively.
• Figure 3: Analysis of residual constraint. In the above example, consider the albedo estimation of point $\mathbf{x}$. (a) when querying incident radiance $L_i(x,\omega_i)$, we show three possibilities of the sampled directions. $\omega_3$ does not hit scene geometry and queries the environment. $\omega_1$ and $\omega_2$ hits the scene at $\mathbf{y_1},\mathbf{y_2}$ respectively, and will query the radiance cache $\mathbf{R}_k$. Note that $\mathbf{y_1}$ can be observed from several cameras, so the cached radiance $\mathbf{R}_k(\mathbf{y_1},-\omega_1)$ may be relatively accurate, in contrast to $\mathbf{R}_k(\mathbf{y_2},-\omega_2)$. (b) in this case, the residual constraint can be useful to correct $\mathbf{R}_k(\mathbf{y_2},-\omega_2)$ by rendering $L_o(\mathbf{y_2},-\omega_2)$ with Equation \ref{['eq:render_incmesh']}, which in turns queries incident radiance from more directly observable regions (like $\mathbf{x}$ and $\mathbf{y}_1$) towards $\mathbf{y_2}$. As a result, the cached value $\mathbf{R}_k(\mathbf{y_2},-\omega_2)$ will be more accurate, which benefits the material estimation of $\mathbf{x}$.
• Figure 4: Qualitative results of albedo estimation. Thanks to our RotLight setup, our method is able to better factor out environment illumination from albedo, resulting in more consistent and accurate estimations. Numbers are PSNR and SSIM metrics.
• Figure 5: Ambient occlusion rendering. With the proxy mesh, our ambient occlusion rendering is smoother and more accurate.