Sketch-guided Cage-based 3D Gaussian Splatting Deformation

TL;DR

This work tackles fine-grained deformation of 3D Gaussian Splatting by introducing a cage-based deformation framework guided by Neural Jacobian Fields and semantically informed diffusion priors. The method enables intuitive edits from a single-view silhouette sketch while preserving rendering quality through a differentiable cage deformation, NJF control, ControlNet, and 3D-aware SDS. It demonstrates precise deformation across diverse objects, supports animation via keyframes, and shows improvements over prior GS editing approaches in both qualitative and quantitative assessments. The combination of geometric regularization and diffusion-based semantic guidance provides a practical pathway to controllable, view-consistent 3D GS editing at scale.

Abstract

3D Gaussian Splatting (GS) is one of the most promising novel 3D representations that has received great interest in computer graphics and computer vision. While various systems have introduced editing capabilities for 3D GS, such as those guided by text prompts, fine-grained control over deformation remains an open challenge. In this work, we present a novel sketch-guided 3D GS deformation system that allows users to intuitively modify the geometry of a 3D GS model by drawing a silhouette sketch from a single viewpoint. Our approach introduces a new deformation method that combines cage-based deformations with a variant of Neural Jacobian Fields, enabling precise, fine-grained control. Additionally, it leverages large-scale 2D diffusion priors and ControlNet to ensure the generated deformations are semantically plausible. Through a series of experiments, we demonstrate the effectiveness of our method and showcase its ability to animate static 3D GS models as one of its key applications.

Figures (9)

• Figure 1: Deformations of 3D Gaussian splats (GS) using our sketch-guided deformation method. Given a 3D GS scene, the user can deform the 3D GS by drawing a deformed silhouette sketch of a single view. (A) and (B) show examples with synthetic data and real-world large-scale data, respectively. We can also produce an animation of a of 3D GS by few ($\geq 2$) keyframe sketches, as shown in (C).
• Figure 2: Overview of our method. We start with a 3D GS model (A). Users can select one specific sketch view (B) to draw an edited silhouette sketch (C). This user-drawn sketch (C) and rendering from the chosen view (B) are fed into ControlNet zhang2023adding resulting in a deformed reference image (D). We then optimize the cage of the source 3D GS model (A) using two losses: (1) a silhouette loss $L_{sil}$ between the silhouette (E) of the deformed GS model (F) and the silhouette of the generated reference image (D), and (2) a 3D-aware SDS loss from $4$ random views (G) conditioned on the reference image (D).
• Figure 3: A simplified 2D illustration of cage-based (in this case, a triangle) deformation of Gaussian splats. $S$ and $S'$ are the original and deformed cage, respectively. $\mu_i$ and $\Sigma_i$ are the centroid and covariance of the original Gaussian splats, and $\mu_i'$ and $\Sigma_i'$ of the deformed ones. $f_S(\mu_i)$ is the cage interpolation function and $J_f(\mu_i)$ is the Jacobian matrix of the interpolation function $f$.
• Figure 4: Controlling Cages via decomposed Neural Jacobian Fields. Instead of optimizing the NJF of the cage $J_i$, we optimized the rotation component $R_i$ and the stretch component $S_i$ of the NJF's transformation respectively. The deformed NJF can be computed easily by applying the transformation to the original NJF. Finally, the deformed cage was obtained by solving the Poisson equation.
• Figure 5: Comparison of using Cage or MLP+HexPlane ren2023dreamgaussian4d to represent deformation of the $3D$ Gaussians. MLP+Hexplane can cause severe fuzzy rendering of the deformed 3D GS because of the lack of geometry regularization. Our cage-based method produces almost lossless deformation regarding the rendering visual quality.