Gaussian Frosting: Editable Complex Radiance Fields with Real-Time Rendering

TL;DR

Gaussian Frosting addresses the challenge of editing complex radiance fields while maintaining high-quality real-time rendering. It combines a mesh-based base surface obtained via SuGaR with an adaptive frosting layer of Gaussians that thickens around fuzzy materials and near-surface volumetrics, and introduces a barycentric, cell-local parameterization to keep Gaussians inside the layer during deformation. The method automatically derives an optimal Poisson reconstruction depth and densifies Gaussians within the frosting, enabling efficient editing, animation, and compositing with improved rendering of hair, grass, and other fine materials. Experiments on synthetic and real scenes show Frosting outperforms purely surface-based methods and matches or exceeds vanilla Gaussian Splatting in rendering quality, while offering interactive editing and a browser-based viewer.

Abstract

We propose Gaussian Frosting, a novel mesh-based representation for high-quality rendering and editing of complex 3D effects in real-time. Our approach builds on the recent 3D Gaussian Splatting framework, which optimizes a set of 3D Gaussians to approximate a radiance field from images. We propose first extracting a base mesh from Gaussians during optimization, then building and refining an adaptive layer of Gaussians with a variable thickness around the mesh to better capture the fine details and volumetric effects near the surface, such as hair or grass. We call this layer Gaussian Frosting, as it resembles a coating of frosting on a cake. The fuzzier the material, the thicker the frosting. We also introduce a parameterization of the Gaussians to enforce them to stay inside the frosting layer and automatically adjust their parameters when deforming, rescaling, editing or animating the mesh. Our representation allows for efficient rendering using Gaussian splatting, as well as editing and animation by modifying the base mesh. We demonstrate the effectiveness of our method on various synthetic and real scenes, and show that it outperforms existing surface-based approaches. We will release our code and a web-based viewer as additional contributions. Our project page is the following: https://anttwo.github.io/frosting/

Figures (8)

• Figure 3: Scene composition. Using mesh editing tools in Blender, we were able to combine various elements from multiple scenes (a) to build a whole new scene (c). We also changed the pose of the characters by using the rigging tool in Blender (b). Similarly to surface-based methods like SuGaR guedon2023sugar, Frosting can be used for editing and compositing scenes, but allows for better rendering of complex volumetric effects and fuzzy materials, such as hair or grass.
• Figure 4: Creating a Layer of Gaussian Frosting. To build our proposed Frosting representation, we start by optimizing a Gaussian Splatting representation using a rendering loss without any additional constraint, to let Gaussians position themselves. We refer to these Gaussians as unconstrained. We then regularize these Gaussians to enforce their alignement with the surface, and extract a mesh that will serve as a basis for the Frosting. Next, we use the misalignment of surface-aligned Gaussians to identify areas where more volumetric rendering is needed, and we build search intervals $J_i$ around the mesh's vertices $\hbox{\boldmath$v_i$}$. Finally, we use the density function of the unconstrained Gaussians to refine the intervals, resulting in a Frosting layer. We finally sample a novel, densified set of Gaussians inside the layer.
• Figure 5: Comparison of meshes extracted by SuGaR from the Shelly dataset without and with our improvement that automatically tunes the octree depth $D$ in Poisson reconstruction depending on the complexity of the scene. Our technique (bottom) drastically reduces surface artifacts for many scenes, such as the holes and the ellipsoidal bumps on the surface when using the default values from guedon2023sugar (top).
• Figure 6: How we define the inner and outer bounds of the Frosting layer. See text in Section \ref{['sec:shifts']}.
• Figure 7: Rendering complex scenes with Frosting. First row: Renderings, Second row: recovered normal maps, Third row: estimated Frosting thickness. Note that the Frosting is thick on fuzzy materials such as the hair and the grass, as expected, and very thin on flat surfaces such as the table on the fourth column.