Temporal Residual Jacobians For Rig-free Motion Transfer

TL;DR

Temporal Residual Jacobians address rig-free motion transfer by learning local spatial and temporal deformations that transfer motion from a stick-figure to unrigged meshes. It jointly trains two networks, integrates changes across space via a differentiable Poisson solve and across time via a neural ODE operating in Jacobian space, with a residual correction mechanism. It does not require canonical templates or rigs during training or inference and can handle long motion sequences across diverse shapes, including non-humanoids. Evaluations on AMASS, DeformingThings4D, and COP3D show improved realism and reduced artifacts compared to VertexODE and NJF baselines. The approach offers a practical pathway to robust motion transfer for diverse, rig-free meshes.

Abstract

We introduce Temporal Residual Jacobians as a novel representation to enable data-driven motion transfer. Our approach does not assume access to any rigging or intermediate shape keyframes, produces geometrically and temporally consistent motions, and can be used to transfer long motion sequences. Central to our approach are two coupled neural networks that individually predict local geometric and temporal changes that are subsequently integrated, spatially and temporally, to produce the final animated meshes. The two networks are jointly trained, complement each other in producing spatial and temporal signals, and are supervised directly with 3D positional information. During inference, in the absence of keyframes, our method essentially solves a motion extrapolation problem. We test our setup on diverse meshes (synthetic and scanned shapes) to demonstrate its superiority in generating realistic and natural-looking animations on unseen body shapes against SoTA alternatives. Supplemental video and code are available at https://temporaljacobians.github.io/ .

Figures (6)

• Figure 1: Given a stick figure dance motion (top-right), Temporal Residual Jacobians retarget the animation to unseen, unrigged meshes (top-left) across time, producing realistic motion dynamics. Please refer to the supplemental webpage for videos. Our method can be trained on limited data, does not require rigged models or skinning weights during training or inference, and does not assume paired sequences or registration to any canonical template mesh. The method was trained on other bodyshapes: no target characters were seen during training. All results in the paper and supplemental material were obtained with automatic feature correspondences and without any postprocessing or smoothing applied.
• Figure 2: Method overview. Starting from input stick figure motion ($\{M_i\}$) and a target body shape ($X_0$), Residual Temporal Jacobians makes local predictions, using primary $f_P$ and residual $f_R$ MLPs, to predict spatial and temporal changes to per-triangle Jacobians. These are then integrated in space, via a Poisson solve, and in time, via numerical Euler stepping, to predict motion dynamics at time frame $t$. These two learnable modules are trained simultaneously with only direct object-level supervision using a combination of positional and Jacobian losses. We use the ground-truth meshes from AMASS to supervise the predictions. The time $t$ input is positionally encoded.
• Figure 3: Generalization across bodyshapes. We show results of different motion transfers on meshes found in-the-wild (blue), FAUST scans (pink) and Mixamo characters (green). We observe a smooth motion consistent with the target geometry in each case. Please see supplemental materials.
• Figure 4: Generalization across shapes with very sparse training sets. Here, we show motion transfer from two animal sequences (in yellow) sampled from the DeformingThings4D dataset def4d, to animal meshes found in the wild (in blue). Our method was trained on only two sequences from this dataset and yet generates plausible motion transfer to unseen shapes. Rigs were not available to our algorithm at training and/or test time. (Note: blue sequences have been slightly globally rotated for visibility.)
• Figure 5: Motion transfer from COP3D dataset. We train on only four sequences of dogs obtained from the COP3D dataset sinha2023common, which are monocular video recordings of animal motion, and transfer the observed regressed motion (in yellow) to creature meshes found in the wild (in blue).