PhysMorph-GS: Differentiable Shape Morphing via Joint Optimization of Physics and Rendering Objectives

TL;DR

PhysMorph-GS addresses the rendering gap in differentiable shape morphing by bidirectionally coupling MLS-MPM with 3D Gaussian Splatting through a deformation-aware upsampling bridge that maps sparse particle states $(\mathbf{x}, \mathbf{F})$ to dense Gaussian parameters $(\boldsymbol{\mu}, \boldsymbol{\Sigma})$. Rendering losses on silhouette and depth backpropagate through both means and covariances to deformation gradients, while mass conservation is enforced on a compact set of anchor particles; a multi-pass interleaved optimization injects rendering gradients into physics to avoid collapse into purely physics-driven solutions. The method introduces adaptive subdivision, multi-scale $\mathbf{F}$-field interpolation, and a covariance formulation $\Sigma' = \mathbf{S} \Sigma \mathbf{S}^T$ derived from the deformation gradient, ensuring physically consistent, anisotropic Gaussians that guide inverse morphing. Ablation studies show depth-only supervision yields a ~2.5% improvement in Chamfer distance over physics-only baselines, with additional rendering channels boosting structural fidelity at the cost of higher particle density. Overall, PhysMorph-GS provides a differentiable, mass-preserving bridge that enables inverse design from image-space supervision in morphing tasks with large deformations and thin structures, advancing the practicality of physics-aware neural rendering for shape optimization.

Abstract

Shape morphing with physics-based simulation naturally supports large deformations and topology changes, but existing methods suffer from a "rendering gap": nondifferentiable surface extraction prevents image losses from directly guiding physics optimization. We introduce PhysMorph-GS, which couples a differentiable material point method (MPM) with 3D Gaussian splatting through a deformation-aware upsampling bridge that maps sparse particle states (x, F) to dense Gaussians (mu, Sigma). Multi-modal rendering losses on silhouette and depth backpropagate along two paths, from covariances to deformation gradients via a stretch-based mapping and from Gaussian means to particle positions. Through the MPM adjoint, these gradients update deformation controls while mass is conserved at a compact set of anchor particles. A multi-pass interleaved optimization scheme repeatedly injects rendering gradients into successive physics steps, avoiding collapse to purely physics-driven solutions. On challenging morphing sequences, PhysMorph-GS improves boundary fidelity and temporal stability over a differentiable MPM baseline and better reconstructs thin structures such as ears and tails. Quantitatively, our depth-supervised variant reduces Chamfer distance by about 2.5 percent relative to the physics-only baseline. By providing a differentiable particle-to-Gaussian bridge, PhysMorph-GS closes a key gap in physics-aware rendering pipelines and enables inverse design directly from image-space supervision.

Figures (12)

• Figure 1: Pipeline overview.(1) Deformation control: Differentiable MPM evolves anchor particles (blue spheres) under log-based grid mass loss $\mathcal{L}_{\text{mass}}$ to maintain material conservation. (2) Surface-aware upsampling: Subdivision upsampling spawns child particles (cyan) in proportion to local deformation magnitude $|\det(\mathbf{F}) - 1|$, upsampling from sparse anchors to a dense set of render particles. A multi-scale $\mathbf{F}$-field then interpolates deformation gradients from coarse anchors to fine render particles with edge-aware regularization, and polar decomposition constructs anisotropic Gaussian covariances $\Sigma' = \mathbf{S}\Sigma\mathbf{S}^T$. (3) Differentiable rendering and gradient fusion: 3DGS renders from the Gaussian parameters and is supervised by silhouette and depth losses ($\mathcal{L}_\alpha, \mathcal{L}_\text{d}$). These rendering gradients are combined with physics gradients via PCGrad and magnitude normalization, and are injected back into the next physics pass.
• Figure 2: Subdivision upsampling process (left to right).(a) Original shape: Initial anchors (8,918 particles) represent the coarse geometry. (b) Parent selection: 99 parents are chosen based on high local deformation, visualized by color. (c) Parent–child relationships: For clarity, we show only the top 20 high-deformation parents (red) and their associated children (orange), highlighting how subdivision concentrates new particles around deformation hotspots. (d) Upsampled children: All 8,811 children are distributed near their parent anchors according to local spacing and jitter. (e) After upsampling (combined): The final point cloud merges anchors and children (17,729 total), resulting in adaptive spatial resolution concentrated at regions of greatest deformation.
• Figure 3: Particle distribution after subdivision (left: $x$, middle: $y$, right: $z$). The adaptive scheme generates a spatially continuous and dense particle field. By concentrating resolution in high-deformation regions, the method ensures smooth coverage without clustering or sparsity artifacts (see Fig. \ref{['fig:upsampling_pipeline']}).
• Figure 4: Multi-scale F-field statistics. Anisotropy and total deformation distributions for the morphing task are set by stability-driven loss design and by the physical material parameters (Young’s modulus $E = 1.4 \times 10^5$, Poisson’s ratio $\nu = 0.3$). The control deformation gradient induces pronounced anisotropy, illustrating how surface-driven constraints directly shape the deformation.
• Figure 5: Rendering loss evolution. Top: $\mathcal{L}_{\alpha}$ localizes errors along shape boundaries ($t=0,10,40$). Bottom: depth hits visualize $\mathcal{L}_{\text{shrink}}$ activation for interior pruning. The combined supervision ensures clean silhouettes and removes internal artifacts as morphing progresses.