TutteNet: Injective 3D Deformations by Composition of 2D Mesh Deformations

TL;DR

TutteNet introduces a principled method for 3D injective deformations by composing multiple 2D mesh deformations derived from Tutte's embedding. Each layer lifts a 2D injective map into 3D as a prismatic transformation, and a sequence of such layers yields a 3D injective deformation $f_\theta$ that is differentiable and has an explicit inverse. The approach combines geometric stability of mesh-based deformations with deep-learning optimization, enabling accurate NeRF and SDF deformations without self-intersections and with tractable Jacobian computations. Compared to purely neural or non-injective methods, TutteNet achieves higher fidelity deformations, robust optimization, and broad applicability to learned deformation spaces from SMPL data and beyond.

Abstract

This work proposes a novel representation of injective deformations of 3D space, which overcomes existing limitations of injective methods: inaccuracy, lack of robustness, and incompatibility with general learning and optimization frameworks. The core idea is to reduce the problem to a deep composition of multiple 2D mesh-based piecewise-linear maps. Namely, we build differentiable layers that produce mesh deformations through Tutte's embedding (guaranteed to be injective in 2D), and compose these layers over different planes to create complex 3D injective deformations of the 3D volume. We show our method provides the ability to efficiently and accurately optimize and learn complex deformations, outperforming other injective approaches. As a main application, we produce complex and artifact-free NeRF and SDF deformations.

Figures (7)

• Figure 1: Elastically deforming a NeRF nerf based on user-designated positioning of the head (turned) tail (bent) and body (lowered), and optimizing the degrees of freedom of TutteNet to minimize the elastic energy of the deformation. TutteNet guarantees an injective (1-to-1) deformation of the ambient 3D space surrounding the T-Rex, ensuring the NeRF is rendered correctly without artifacts by enabling "pulling back" points and view directions from deformed space. Each layer within TutteNet views the T-Rex over a different 2D plane (in this case, alternating between the 3 main axes in a tri-plane manner). In each layer, the T-Rex is enveloped with a regular 2D mesh of the unit square (top row). The 2D mesh is deformed using the layer's optimizeable parameters which define a Tutte's embedding tutte1963drawfloater2003one (bottom row). This defines an injective 2D piecewise-linear map, which can be applied to the 3D T-Rex, without modifying the normal direction to the plane, resulting in an injective 3D deformation $\Phi^i$. Composition of these layers yields the final expressive 3D injective deformation.
• Figure 2: Our representation of injective 3D deformations, visualized for the process of mapping a given point $\mathbf{p}$ inside the volume, for two-layer TutteNet, $i\in\left\{1,2\right\}$. Left: the (learnable) parameters $\theta^i$, consisting of the mesh-Laplacian $L^i$ and the boundary conditions $\mathbf{b}^i$, define a 2D deformation $\Psi^i$ of the square mesh $\mathbf{M}$, through Tutte's embedding tutte1963draw. $\Psi^i$ is embedded in 3D to the local coordinates $\mathbf{R}^i$ to define a 3D deformation, $\Phi^i$. Right: given a point $\mathbf{p}$, it is projected to the local coordinates of $\Phi^1$, landing on triangle $\mathbf{t}^1$. $\Phi^1$ defines an affine map over the infinite prism of $\mathbf{t}^1$ (represented in blue with dotted lines), mapping $\mathbf{p}$ to $\Phi^1(\mathbf{p})$. The resulting $\Phi^1\left (\mathbf{p}\right )$ is projected onto the local coordinates of $\Psi^2$, landing on triangle $\mathbf{t}^2$, from which it is mapped by the affine map $\Phi^2$ defined over the infinite prism of $\mathbf{t}^2$.
• Figure 3: Comparing NeRF deformation methods. We minimize the elastic deformation energy of NeRFs under user-specified constraints (left, in green) and compare the visual quality of our results with other techniques. Non-injective methods such as NeRF-Editing nerfediting and Deforming-NeRF deformingRFCages lead to non-injective deformations due to internal inversions and intersections, in turn leading to visible artifacts. SPIDR liang2022spidr relies on a hybrid SDF/point cloud representation, leading to degradation in detail (T-Rex teeth) as well non-injective artifacts (tractor). We additionally compare to the only other injective method that is applicable for this experiment, NeuralODE chen2018neural whose injectivity avoids visual artifacts, but causes geometric artifacts such as squashing the T-Rex's tail and the robot's eye.
• Figure 4: Elastic deformations of NeRFs. Our method guarantees injectivity and enables complex deformations, like tying a loop on the microphone's cord.
• Figure 5: Elastic deformations of in-the-wild NeRFs. Our method is applicable to produce deformation of lower-quality NeRFs, captured using a phone app (Record3D).