GSDeformer: Direct, Real-time and Extensible Cage-based Deformation for 3D Gaussian Splatting

TL;DR

GSDeformer presents a real-time, cage-based deformation framework for 3D Gaussian Splatting (3DGS) that operates directly on trained vanilla 3DGS models without architectural changes or retraining. It constructs a proxy point cloud from Gaussian primitives, applies cage-based deformation via mean value coordinates, and transfers the resulting transformation back to the Gaussians, with a splitting step to manage bending artifacts. A novel render-reconstruct-simplify cage-building pipeline (including depth rendering and T-SDF integration) enables automatic cage extraction for 3DGS and its variants such as 2DGS, ensuring robust, high-quality deformation across extreme cases. The approach achieves real-time performance (around 60 FPS for deformation and >200 FPS rendering), demonstrates superior deformation quality over existing methods, and integrates easily with other 3DGS editing techniques and variants, making it broadly practical for animation, composition, and editing workflows.

Abstract

We present GSDeformer, a method that enables cage-based deformation on 3D Gaussian Splatting (3DGS). Our approach bridges cage-based deformation and 3DGS by using a proxy point-cloud representation. This point cloud is generated from 3D Gaussians, and deformations applied to the point cloud are translated into transformations on the 3D Gaussians. To handle potential bending caused by deformation, we incorporate a splitting process to approximate it. Our method does not modify or extend the core architecture of 3D Gaussian Splatting, making it compatible with any trained vanilla 3DGS or its variants. Additionally, we automate cage construction for 3DGS and its variants using a render-and-reconstruct approach. Experiments demonstrate that GSDeformer delivers superior deformation results compared to existing methods, is robust under extreme deformations, requires no retraining for editing, runs in real-time, and can be extended to other 3DGS variants. Project Page: https://jhuangbu.github.io/gsdeformer/

Figures (15)

• Figure 1: Overview of our cage-building algorithm. Given an object, our method renders depth image from it, performs T-SDF integration, surface extraction and space carving to produce a solid voxel grid. The voxel grid is then simplified using morphological closing operator, meshed using marching cube, and decimated to obtain the final cage mesh.
• Figure 2: Overview of our deformation algorithm. The deformation process is shown in 2D for clarity. For deformation, 3DGS Gaussians are converted to ellipsoids represented using points (the proxy point cloud). Proxy points are deformed using cage-based deformation and split if their axes are bent. Finally, deformed points are used to infer transformations for the Gaussians. For more details, please refer to the pseudo-code in Appendix \ref{['sect:apptx-pseudocode']}.
• Figure 3: The splitting process. Our method fixes the ill-formed bent Gaussian by splitting the Gaussian before MVC deformation, leading to well-formed Gaussians and, thus, reasonable transforms.
• Figure 4: Comparison of cage building algorithm. We present the raw voxel grids and the produced final cages for comparison. Red circles and cyan boxs marks defects. Note that our method generates cleaner voxel grids and smoother final cages for both 3DGS and 2DGS. The marching cube(MC) baseline generates overly dense mesh for 3DGS scenes and completely fails on 2DGS scenes. Our smoothing process enabled it to produce coherent results, but artifacts remain.
• Figure 5: Comparison of methods on selected objects. Red boxes indicate zoomed areas; cyan circles marks defects. Not having defect marks indicates satisfactory results. Our approach is the only method that performs well across all cases. For more results, refer to our supplementary video.