Subsurface Scattering for 3D Gaussian Splatting

TL;DR

This work proposes a framework for optimizing an object's shape together with the radiance transfer field given multi-view OLAT (one light at a time) data and achieves comparable or better results at a fraction of optimization and rendering time while enabling detailed control over material attributes.

Abstract

3D reconstruction and relighting of objects made from scattering materials present a significant challenge due to the complex light transport beneath the surface. 3D Gaussian Splatting introduced high-quality novel view synthesis at real-time speeds. While 3D Gaussians efficiently approximate an object's surface, they fail to capture the volumetric properties of subsurface scattering. We propose a framework for optimizing an object's shape together with the radiance transfer field given multi-view OLAT (one light at a time) data. Our method decomposes the scene into an explicit surface represented as 3D Gaussians, with a spatially varying BRDF, and an implicit volumetric representation of the scattering component. A learned incident light field accounts for shadowing. We optimize all parameters jointly via ray-traced differentiable rendering. Our approach enables material editing, relighting and novel view synthesis at interactive rates. We show successful application on synthetic data and introduce a newly acquired multi-view multi-light dataset of objects in a light-stage setup. Compared to previous work we achieve comparable or better results at a fraction of optimization and rendering time while enabling detailed control over material attributes. Project page https://sss.jdihlmann.com/

Figures (11)

• Figure 1: SSS GS -- We propose photorealistic real-time relighting and novel view synthesis of subsurface scattering objects. We learn to reconstruct the shape and translucent appearance of an object within the Gaussian Splatting framework. To do so we leverage our newly created multi-view multi-light dataset of synthetic and real-world objects acquired in a light-stage setup. The object is decomposed in a PBR fashion allowing for easy material editing and relighting. For a trailer visit our project page at https://sss.jdihlmann.com/.
• Figure 2: Subsurface Scattering Pipeline - Our method implicitly models the subsurface scattering appearance of an object and combines it with an explicit surface appearance model. The object is represented as a set of 3D Gaussians, consisting of geometry and appearence properties. We ultilize a small MLP to evaluate the subsurface scattering residual given the view and light direction and a subset of properties for each Gaussian. Further, we evaluate the incident light for each Gaussian as a joint task within the same MLP given the visibility supervised by ray-tracing. Based on the computed properties we accumulate and rasterize each property on the image plane in a deferred shading pipeline. We evaluate the diffuse and specular color with a BRDF model for every pixel in image space and combine it with the SSS residual to get the final color of the object.
• Figure 3: Results of Decomposition -- showing two different views with different light directions. Further, the decomposition of PBR parameters is shown. The first two objects shown are synthetic while the lower two are scanned real-world world objects.
• Figure 4: Editing results, showcasing PBR based edits such as (roughness / metalness / base color) as well as method specific properties (subsurfaceness / residual color). The latter highlights editing only possible with this method. The rightmost column shows light positions not sampled from the light stage.
• Figure 5: Limits of Relightable 3D Gaussians (left) -- While Relightable 3D Gaussians can reproduce view and illumination-dependent reflections they fail to properly relight subsurface scattering objects. Deferred Shading (right) -- allows us to evaluate the surface reflectance for each rendered pixel instead of per Gaussian. This way, specular highlights are rendered with crisper detail.