GS-ROR$^2$: Bidirectional-guided 3DGS and SDF for Reflective Object Relighting and Reconstruction

TL;DR

<3-5 sentence high-level summary> GS-ROR^2 tackles the challenge of relighting and reconstructing reflective 3D objects from multi-view imagery by marrying 3D Gaussian Splatting with a lightweight SDF-based prior. The method introduces mutual, rendering-free supervision between deferred Gaussian attributes and an auxiliary SDF (via depth and normal) to constrain geometry without expensive volume rendering, plus an adaptive pruning strategy to mitigate floater artifacts. A GS-guided SDF enhancement step upscales and fine-tunes a low-resolution SDF (TensoSDF) using the Gaussian-derived normals, delivering high-quality meshes while keeping training time modest. The approach achieves robust relighting and geometric fidelity, offering real-time rendering and competitive performance relative to NeRF-based methods with substantially lower computational costs.</p>

Abstract

3D Gaussian Splatting (3DGS) has shown a powerful capability for novel view synthesis due to its detailed expressive ability and highly efficient rendering speed. Unfortunately, creating relightable 3D assets and reconstructing faithful geometry with 3DGS is still problematic, particularly for reflective objects, as its discontinuous representation raises difficulties in constraining geometries. Volumetric signed distance field (SDF) methods provide robust geometry reconstruction, while the expensive ray marching hinders its real-time application and slows the training. Besides, these methods struggle to capture sharp geometric details. To this end, we propose to guide 3DGS and SDF bidirectionally in a complementary manner, including an SDF-aided Gaussian splatting for efficient optimization of the relighting model and a GS-guided SDF enhancement for high-quality geometry reconstruction. At the core of our SDF-aided Gaussian splatting is the mutual supervision of the depth and normal between blended Gaussians and SDF, which avoids the expensive volume rendering of SDF. Thanks to this mutual supervision, the learned blended Gaussians are well-constrained with a minimal time cost. As the Gaussians are rendered in a deferred shading mode, the alpha-blended Gaussians are smooth, while individual Gaussians may still be outliers, yielding floater artifacts. Therefore, we introduce an SDF-aware pruning strategy to remove Gaussian outliers located distant from the surface defined by SDF, avoiding floater issue. This way, our GS framework provides reasonable normal and achieves realistic relighting, while the mesh from depth is still problematic. Therefore, we design a GS-guided SDF refinement, which utilizes the blended normal from Gaussians to finetune SDF. With this enhancement, our method can further provide high-quality meshes for reflective objects at the cost of 17% extra training time.

Figures (21)

• Figure 1: Overview of our SDF-aided Gaussian Splatting. The architecture of our proposed method consists of two geometry representations (i.e., Gaussian primitive and TensoSDF). In the deferred Gaussian pipeline, the shading parameters (i.e., albedo $\textbf{a}$, roughness $\textbf{r}$ and metallicity $\textbf{m}$), normal and depth are projected to the image plane and alpha blended. The pixel color $C_{\rm gs}$ is calculated using the split-sum approximation and supervised by ground truth color $C_{\rm gt}$. In the TensoSDF, we sample rays originated from camera center $\textbf{o}$ and view direction $\textbf{v}$ and query the SDF value and gradient for each point $p$ along the ray $\textbf{o}+t\textbf{v}$. The normal $\textbf{n}_{\rm sdf}$ and depth $D_{\rm sdf}$ are obtained via volume rendering, which is supervised mutually with the normal $\textbf{n}_{\rm gs}$ the depth $D_{\rm gs}$ from Gaussians. Note no color network is used in the SDF part, and only the geometry attributes are volume rendered.
• Figure 2: The geometry from Gaussian is under-constrained and thus erroneous, while it is much better after utilizing the priors from the SDF.
• Figure 3: Two Gaussians with a minor normal difference are overlapped to model an opaque surface. In the forward rendering, the BRDF values are computed w.r.t. to their own normal and are then alpha blended to form the final rendering, which is equivalent to a broader BRDF lobe, leading to a blurry rendering, eventually. In contrast, in the deferred shading, the BRDF is computed w.r.t. the deferred normal, maintaining the sharpness of reflective objects.
• Figure 4: In this example, the floaters can still be shown in the testing view, even if the mask loss was applied during training. The main reason is that although the floaters are shown outside the mask region in the testing view, they are within the mask region in the training views. Therefore, they can not be masked out by the mask loss.
• Figure 5: The illustration of SDF-aware pruning. We define a narrowing threshold, which is adjusted automatically around the zero-level set. The Gaussians out of the threshold will be pruned. This pruning operation ensures all Gaussians are near the surface and avoids the floaters.