CAGE-GS: High-fidelity Cage Based 3D Gaussian Splatting Deformation

TL;DR

This work addresses deformations of 3D Gaussian Splatting (3DGS) scenes while preserving fine texture. It introduces CAGE-GS, which combines a learned, target-informed cage with a Jacobian-based covariance update to transfer geometry and maintain texture across diverse target representations, including texts, images, point clouds, meshes, and 3DGS models. The method defines Gaussians as $g_i=\oldsymbol{\mu}_i,\boldsymbol{\Sigma}_i,c_i,\alpha_i$ with $\boldsymbol{\Sigma}_i=R S S^{T} R^{T}$, learns a deformed cage $C_{s\to t}$ via neural encoders/decoders, and updates Gaussian covariances using $J=\partial \boldsymbol{\mu}'/\partial \boldsymbol{\mu}$ to obtain $\boldsymbol{\Sigma}'= J R S S^{T} R^{T} J^{T}$, ensuring texture fidelity after deformation. Contributions include the cage-based deformation mechanism for 3DGS with automated transfer, the Jacobian-driven covariance adaptation, and demonstrated improvements in deformation quality and efficiency across multiple target types. This framework enables fast, flexible, high-fidelity edits for 3D content creation, editing, and scene manipulation. The approach leverages the strengths of 3DGS representations while introducing a structured deformation space that preserves surface details during shape transfer.

Abstract

As 3D Gaussian Splatting (3DGS) gains popularity as a 3D representation of real scenes, enabling user-friendly deformation to create novel scenes while preserving fine details from the original 3DGS has attracted significant research attention. We introduce CAGE-GS, a cage-based 3DGS deformation method that seamlessly aligns a source 3DGS scene with a user-defined target shape. Our approach learns a deformation cage from the target, which guides the geometric transformation of the source scene. While the cages effectively control structural alignment, preserving the textural appearance of 3DGS remains challenging due to the complexity of covariance parameters. To address this, we employ a Jacobian matrix-based strategy to update the covariance parameters of each Gaussian, ensuring texture fidelity post-deformation. Our method is highly flexible, accommodating various target shape representations, including texts, images, point clouds, meshes and 3DGS models. Extensive experiments and ablation studies on both public datasets and newly proposed scenes demonstrate that our method significantly outperforms existing techniques in both efficiency and deformation quality.

Figures (10)

• Figure 1: Results of CAGE-GS.Top: Our method deforms the source chair to match the target models, supporting multiple target representations, including meshes, point clouds, images, texts and 3DGS. Bottom: Our method is able to control the deformation magnitude through cage interpolation.
• Figure 2: Overview. Our cage-based deformation framework has two modules. In the Cage Prediction Module, it learns the source cage from the source model and the cage offset from the target to generate a deformed cage. In the Gaussian Deforming Module, the deformed cage and optimized mean value coordinates from the source cage are used to deform the positions of the 3D Gaussians while keeping their covariance matrix intact. Finally, we employ the Jacobian matrix to effectively update the covariance matrix of the 3D Gaussians, maintaining high-fidelity texture on the rendered images. Our method supports multiple target representations, including texts, meshes, point clouds, images and 3DGS.
• Figure 3: Deformation results on ShapeNet dataset. We validate our method on the chair, car and table categories from ShapeNet chang2015shapenet. The results show that our method aligns the source shape with the target shape while preserving the source texture.
• Figure 4: Deformation results on different target representations. The results show that our method can support different target formats, including images, point clouds, meshes and 3DGS model generated with text prompt.
• Figure 5: Deformation results on real-captured data. We validate our method on a real-captured teapot and a mug. The results show that our method can apply deformation transfer to the real-captured models.