RGS-DR: Deferred Reflections and Residual Shading in 2D Gaussian Splatting

TL;DR

RGS-DR tackles glossy inverse rendering by integrating 2D Gaussian splatting with a pixel-deferred shading pipeline and a directional residual refinement to capture unresolved view-dependent effects. It explicitly models geometry, materials, and environment illumination to enable photorealistic rendering, relighting, and scene editing for shiny objects. A G-buffer-based shading system with split-sum IBL, plus a lightweight residual branch, sharpens specular highlights and reduces artifacts, bridging the gap to reconstruction-only methods. Experiments on synthetic and real shiny datasets demonstrate improved specular quality, sharper environment maps, and robust editing capabilities, while acknowledging limitations in modeling multi-bounce light transport.

Abstract

In this work, we address specular appearance in inverse rendering using 2D Gaussian splatting with deferred shading and argue for a refinement stage to improve specular detail, thereby bridging the gap with reconstruction-only methods. Our pipeline estimates editable material properties and environment illumination while employing a directional residual pass that captures leftover view-dependent effects for further refining novel view synthesis. In contrast to per-Gaussian shading with shortest-axis normals and normal residuals, which tends to result in more noisy geometry and specular appearance, a pixel-deferred surfel formulation with specular residuals yields sharper highlights, cleaner materials, and improved editability. We evaluate our approach on rendering and reconstruction quality on three popular datasets featuring glossy objects, and also demonstrate high-quality relighting and material editing.

Figures (15)

• Figure 1: RGS-DR delivers high-quality inverse rendering by disentangling scene geometry, material properties, and illumination, enabling photorealistic rendering, relighting, and scene editing.
• Figure 2: Our rendering pipeline consists of three passes. The geometry pass produces screen-space diffuse color $\mathcal{I}_d$, specular tint $S$, roughness $M$, normals $N$, and low-dimensional features $K$, which feed into the subsequent passes. The lighting pass employs a cube mipmap to model environmental light for shading as described in jiang2024gshader. Meanwhile, the residual pass uses a spherical-mip-based directional encoding (inspired by zhang2024refgs) along with a shallow MLP $f_\text{res}$ to predict view-dependent effects not captured by the lighting pass.
• Figure 3: Comparison of the rendering quality of our method against Ref-GS zhang2024refgs, 3DGS-DR ye2024gsdr and GaussianShader jiang2024gshader.
• Figure 4: Outputs of our model including renderings, material properties, residuals and surface normals for scenes from the Shiny Synthetic and Glossy Synthetic datasets. The residuals are amplified for visualization purposes.
• Figure 5: Comparison of estimated environment maps of synthetic scenes from the Shiny Synthetic dataset verbin2022refnerf.