DeMapGS: Simultaneous Mesh Deformation and Surface Attribute Mapping via Gaussian Splatting

TL;DR

DeMapGS addresses the lack of topological coherence in Gaussian splatting by anchoring structured Gaussian splats to a deformable mesh, enabling joint optimization of geometry and splat parameters. It introduces gradient diffusion and a 2DGS/3DGS alternating rendering pipeline to support robust, large-step deformation while preserving photorealistic rendering, and it extracts high-quality diffuse, normal, and displacement maps suitable for standard graphics workflows. The approach achieves state-of-the-art mesh reconstruction quality and supports downstream editing and cross-object manipulation via a shared surface, complemented by a texture refinement step and GPU-accelerated map rasterization. Together, these contributions bridge structured mesh representations with Gaussian splatting, enabling practical editing, rendering, and cross-object operations within existing graphics pipelines.

Abstract

We propose DeMapGS, a structured Gaussian Splatting framework that jointly optimizes deformable surfaces and surface-attached 2D Gaussian splats. By anchoring splats to a deformable template mesh, our method overcomes topological inconsistencies and enhances editing flexibility, addressing limitations of prior Gaussian Splatting methods that treat points independently. The unified representation in our method supports extraction of high-fidelity diffuse, normal, and displacement maps, enabling the reconstructed mesh to inherit the photorealistic rendering quality of Gaussian Splatting. To support robust optimization, we introduce a gradient diffusion strategy that propagates supervision across the surface, along with an alternating 2D/3D rendering scheme to handle concave regions. Experiments demonstrate that DeMapGS achieves state-of-the-art mesh reconstruction quality and enables downstream applications for Gaussian splats such as editing and cross-object manipulation through a shared parametric surface.

Figures (11)

• Figure 1: (a) Gradients of 2D splat positions lie in the plane perpendicular to the view direction; (b) In concave regions, splats are only visible from frontal views, yielding weak gradients along the surface normal. (c) Illustration of variation in projected 2D covariance for 2DGS and 3DGS under changes in the camera-space depth of the Gaussian mean.
• Figure 2: Detail optimization stages. Each stage employs different loss terms tailored to its purpose, as shown.
• Figure 3: (a) Gradient flow from splats to vertex $\bm{v}_i$ (omitting $M$): gradients are first propagated to vertices on the attached face, then diffused to $\bm{v}_i$ via weighted averaging. (b) Vertex realignment: target position is computed as a weighted average of neighboring splat centers.
• Figure 4: OpenGL-rendered geometry of reconstructed meshes. Deformation-based methods fail to fully deform to the target shape or produce noisy geometry. Reconstruction-based methods tend to preserve the overall structure but may exhibit noise or distortion. Our method, based on 2DGS rendering, closely resembles the 2DGS baseline. Due to tessellation and normal maps, our results exhibit a smoother appearance.
• Figure 5: OpenGL-rendered results with and without texture. 2DGS relies on per-vertex colors, resulting in visibly blurred appearance. SuGaR uses diffuse maps, which lead to visually appealing results. However, the underlying geometry suffers from artifacts, resulting in unstable performance. Our method achieves overall cleaner and sharper renderings by combining structured geometry with high-quality diffuse map extraction. Models © matousekfoto (Tree), Geoffrey Marchal (Bust), hullo (Buddha), pigfinite (Book), 3D Master (Box), Barbucha Studio (Cat).