Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PhyGaP: Physically-Grounded Gaussians with Polarization Cues

Jiale Wu, Xiaoyang Bai, Zongqi He, Weiwei Xu, Yifan Peng

Abstract

Recent advances in 3D Gaussian Splatting (3DGS) have demonstrated great success in modeling reflective 3D objects and their interaction with the environment via deferred rendering (DR). However, existing methods often struggle with correctly reconstructing physical attributes such as albedo and reflectance, and therefore they do not support high-fidelity relighting. Observing that this limitation stems from the lack of shape and material information in RGB images, we present PhyGaP, a physically-grounded 3DGS method that leverages polarization cues to facilitate precise reflection decomposition and visually consistent relighting of reconstructed objects. Specifically, we design a polarimetric deferred rendering (PolarDR) process to model polarization by reflection, and a self-occlusion-aware environment map building technique (GridMap) to resolve indirect lighting of non-convex objects. We validate on multiple synthetic and real-world scenes, including those featuring only partial polarization cues, that PhyGaP not only excels in reconstructing the appearance and surface normal of reflective 3D objects (~2 dB in PSNR and 45.7% in Cosine Distance better than existing RGB-based methods on average), but also achieves state-of-the-art inverse rendering and relighting capability. Our code will be released soon.

PhyGaP: Physically-Grounded Gaussians with Polarization Cues

Abstract

Recent advances in 3D Gaussian Splatting (3DGS) have demonstrated great success in modeling reflective 3D objects and their interaction with the environment via deferred rendering (DR). However, existing methods often struggle with correctly reconstructing physical attributes such as albedo and reflectance, and therefore they do not support high-fidelity relighting. Observing that this limitation stems from the lack of shape and material information in RGB images, we present PhyGaP, a physically-grounded 3DGS method that leverages polarization cues to facilitate precise reflection decomposition and visually consistent relighting of reconstructed objects. Specifically, we design a polarimetric deferred rendering (PolarDR) process to model polarization by reflection, and a self-occlusion-aware environment map building technique (GridMap) to resolve indirect lighting of non-convex objects. We validate on multiple synthetic and real-world scenes, including those featuring only partial polarization cues, that PhyGaP not only excels in reconstructing the appearance and surface normal of reflective 3D objects (~2 dB in PSNR and 45.7% in Cosine Distance better than existing RGB-based methods on average), but also achieves state-of-the-art inverse rendering and relighting capability. Our code will be released soon.
Paper Structure (38 sections, 28 equations, 16 figures, 5 tables)

This paper contains 38 sections, 28 equations, 16 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reflective Object Reconstruction and Inverse Rendering.
  4. Polarization-Based Vision and 3D Reconstruction.
  5. Preliminaries: Ref-Gaussian
  6. 2D Gaussian Splatting
  7. Deferred Rendering (DR)
  8. Method
  9. PolarDR: Polarimetric Deferred Rendering
  10. GridMap: Self-Occlusion-Aware Environment Map Building
  11. Training Strategies
  12. Experiments
  13. Implementation Details
  14. Baselines.
  15. Datasets.
  16. ...and 23 more sections

Figures (16)

  • Figure 1: We propose PhyGaP, a physically-grounded 3DGS method that (a) takes full or partial polarization information as input, (b) accurately reconstructs the shape and physical attributes of glossy object, (c) achieves decomposed rendering of object appearance (top), diffuse reflection (mid), and specular reflection (bottom), as well as (d) enables robust and realistic relighting with natural reflection. Results in this visualization are from our captured buddha scene.
  • Figure 2: Overview of the PhyGaP pipeline. We represent physically-grounded attributes such as roughness, albedo, IoR and surface normal with 2DGS, and render them into Stokes values via the PolarDR process. Furthermore, we design the GridMap technique to tackle self-occlusion of nonconvex objects. By utilizing polarization cues, we achieve accurate, explicit and disentangled representation of object albedo, diffuse reflection and specular reflection in PhyGaP.
  • Figure 3: Overview of GridMap. (a) We sample a set of anchor cameras on the object bounding box. (b) For each anchor camera, we ray trace in all directions to blend object color with the global environment map. (c) The resulting local environment map enables more accurate diffuse irradiance computation.
  • Figure 4: Visualization of reconstructed surface normal for synthetic (snail) and real-world (frog) objects.
  • Figure 5: Qualitative comparison on estimated environment maps.
  • ...and 11 more figures