Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DeferredGS: Decoupled and Editable Gaussian Splatting with Deferred Shading

Tong Wu, Jia-Mu Sun, Yu-Kun Lai, Yuewen Ma, Leif Kobbelt, Lin Gao

TL;DR

DeferredGS tackles editing limitations in Gaussian splatting by decoupling geometry, texture, and lighting. It introduces per-Gaussian texture attributes and a normal direction learned through normal field distillation from an SDF-based network, then uses deferred shading with a learnable environment map for realistic relighting. The approach enables geometry and texture editing as well as relighting under novel illumination while delivering competitive novel-view synthesis results and real-time performance. Limitations include challenges with shadows, suggesting future work on visibility-aware rendering and multi-illumination capture to further improve robustness.

Abstract

Reconstructing and editing 3D objects and scenes both play crucial roles in computer graphics and computer vision. Neural radiance fields (NeRFs) can achieve realistic reconstruction and editing results but suffer from inefficiency in rendering. Gaussian splatting significantly accelerates rendering by rasterizing Gaussian ellipsoids. However, Gaussian splatting utilizes a single Spherical Harmonic (SH) function to model both texture and lighting, limiting independent editing capabilities of these components. Recently, attempts have been made to decouple texture and lighting with the Gaussian splatting representation but may fail to produce plausible geometry and decomposition results on reflective scenes. Additionally, the forward shading technique they employ introduces noticeable blending artifacts during relighting, as the geometry attributes of Gaussians are optimized under the original illumination and may not be suitable for novel lighting conditions. To address these issues, we introduce DeferredGS, a method for decoupling and editing the Gaussian splatting representation using deferred shading. To achieve successful decoupling, we model the illumination with a learnable environment map and define additional attributes such as texture parameters and normal direction on Gaussians, where the normal is distilled from a jointly trained signed distance function. More importantly, we apply deferred shading, resulting in more realistic relighting effects compared to previous methods. Both qualitative and quantitative experiments demonstrate the superior performance of DeferredGS in novel view synthesis and editing tasks.

DeferredGS: Decoupled and Editable Gaussian Splatting with Deferred Shading

TL;DR

DeferredGS tackles editing limitations in Gaussian splatting by decoupling geometry, texture, and lighting. It introduces per-Gaussian texture attributes and a normal direction learned through normal field distillation from an SDF-based network, then uses deferred shading with a learnable environment map for realistic relighting. The approach enables geometry and texture editing as well as relighting under novel illumination while delivering competitive novel-view synthesis results and real-time performance. Limitations include challenges with shadows, suggesting future work on visibility-aware rendering and multi-illumination capture to further improve robustness.

Abstract

Reconstructing and editing 3D objects and scenes both play crucial roles in computer graphics and computer vision. Neural radiance fields (NeRFs) can achieve realistic reconstruction and editing results but suffer from inefficiency in rendering. Gaussian splatting significantly accelerates rendering by rasterizing Gaussian ellipsoids. However, Gaussian splatting utilizes a single Spherical Harmonic (SH) function to model both texture and lighting, limiting independent editing capabilities of these components. Recently, attempts have been made to decouple texture and lighting with the Gaussian splatting representation but may fail to produce plausible geometry and decomposition results on reflective scenes. Additionally, the forward shading technique they employ introduces noticeable blending artifacts during relighting, as the geometry attributes of Gaussians are optimized under the original illumination and may not be suitable for novel lighting conditions. To address these issues, we introduce DeferredGS, a method for decoupling and editing the Gaussian splatting representation using deferred shading. To achieve successful decoupling, we model the illumination with a learnable environment map and define additional attributes such as texture parameters and normal direction on Gaussians, where the normal is distilled from a jointly trained signed distance function. More importantly, we apply deferred shading, resulting in more realistic relighting effects compared to previous methods. Both qualitative and quantitative experiments demonstrate the superior performance of DeferredGS in novel view synthesis and editing tasks.
Paper Structure (24 sections, 10 equations, 10 figures, 3 tables)

This paper contains 24 sections, 10 equations, 10 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Neural Rendering
  4. Neural Surface Reconstruction
  5. Inverse Rendering
  6. Method
  7. Gaussian Splatting and Texture Definition
  8. Normal Field Distillation
  9. Deferred Shading in Gaussian Splatting
  10. Editing Decoupled Gaussians
  11. Relighting
  12. Geometry Editing
  13. Texture Editing
  14. Results and Evaluations
  15. Datasets and Metrics
  16. ...and 9 more sections

Figures (10)

  • Figure 1: Given multi-view images, DeferredGS optimizes a Gaussian splatting representation with decoupled geometry, texture, and lighting. With this decoupled representation, DeferredGS not only allows geometry and texture editing, but can also render the input scene (the 3rd column, bottom row) or the edited scene (the last column, bottom row) under a novel illumination.
  • Figure 2: Overview of DeferredGS. Given multi-view images, DeferredGS builds a decoupled Gaussian splatting representation where auxiliary texture attributes are defined on each Gaussian. To enable successful geometry and appearance separation, we attach an extra normal direction to each Gaussian and optimize it by distilling the normal field from a jointly trained signed distance function. For texture and lighting decomposition, DeferredGS rasterizes geometry and texture attributes into buffer maps and computes shading at the pixel level under the illumination of a learnable environment map with deferred shading.
  • Figure 3: Novel view synthesis comparisons with NeRFactor NeRFactor, TensoIR TensoIR, NDR NvDiffRec, NeRO NeRO, Gaussian Splatting (GS) GS, RelightableGaussian (RGS) RelightableGaussian, and GaussianShader (GShader) GaussianShader. The bottom two rows are from the real Stanford ORB dataset Stanford-ORB.
  • Figure 4: Decomposed result comparisons with NeRFactor NeRFactor, TensoIRTensoIR, NDR NvDiffRec, NeRO NeRO, RelightableGaussian (RGS) RelightableGaussian, and GaussianShader (GShader) GaussianShader. In every two rows, we show decomposed normal and diffuse albedo components by different methods and compare them with the ground truth. Note that GaussianShader only decomposes diffuse color instead of diffuse albedo so its diffuse albedo results are unavailable.
  • Figure 5: Relighting comparisons with NeRFactor NeRFactor, TensoIRTensoIR, NDR NvDiffRec, NeRO NeRO, RelightableGaussian (RGS) RelightableGaussian, and GaussianShader (GShader) GaussianShader. In the first column, we show the input scene and target environment map. Relighting results by different methods and the ground truth are in other columns.
  • ...and 5 more figures