VolTeMorph: Realtime, Controllable and Generalisable Animation of Volumetric Representations

TL;DR

VolTeMorph tackles the challenge of animating volumetric scene representations in real time while remaining artist-friendly and generalisable. The method deforms a canonical volumetric field using a tetrahedral mesh, maps samples between deformed and canonical spaces, and rotates view directions to preserve lighting under motion, enabling both object-level physics deformations and facial avatar animation via Vol3DMM. Key contributions include VolTeMorph itself, Vol3DMM for volumetric faces, a real-time rendering pipeline based on FastNeRF, and extensive experiments showing superior extrapolation and novel-view capability compared with mesh- or implicit-deformation baselines. The work offers practical impact for real-time telepresence, interactive VR/AR experiences, and game/XR content creation, delivering photorealistic deformations with intuitive, off-the-shelf editing and scalable enrolment from limited data.

Abstract

The recent increase in popularity of volumetric representations for scene reconstruction and novel view synthesis has put renewed focus on animating volumetric content at high visual quality and in real-time. While implicit deformation methods based on learned functions can produce impressive results, they are `black boxes' to artists and content creators, they require large amounts of training data to generalise meaningfully, and they do not produce realistic extrapolations outside the training data. In this work we solve these issues by introducing a volume deformation method which is real-time, easy to edit with off-the-shelf software and can extrapolate convincingly. To demonstrate the versatility of our method, we apply it in two scenarios: physics-based object deformation and telepresence where avatars are controlled using blendshapes. We also perform thorough experiments showing that our method compares favourably to both volumetric approaches combined with implicit deformation and methods based on mesh deformation.

Figures (20)

• Figure 1: Overview of the VolTeMorph rendering process. To render a single pixel a ray is cast from the camera centre, through the pixel into the scene in its deformed state. A number of samples are generated along the ray and then each sample is mapped to the canonical space using the deformation $M_j$ of the corresponding tetrahedron $j$ (Section \ref{['sec:tetrahedra']}). The volumetric representation of the scene (Section \ref{['sec:nerf_basics']}) is then queried with the deformed sample position $p'_j$ and the direction of the ray rotated based on the rotation of the $j$-th tetrahedron (Section \ref{['sec:view_dir_rotation']}). The resulting per-sample density and colour values are then integrated using volume rendering (Equation \ref{['eq:rendering_equation']}).
• Figure 2: A point $\bm{p}$ in deformed spaced is mapped to $\bm{\hat{p}}$ in canonical space using barycentric coordinates defined for both the canonical tetrahedron $X = \{\bm{x}_1, \bm{x}_2, \bm{x}_3, \bm{x}_4\}$ as well as the deformed tetrahedron $\hat{X} = \{\hat{\bm{x}}_1, \hat{\bm{x}}_2, \hat{\bm{x}}_3, \hat{\bm{x}}_4\}$.
• Figure 3: Converting a static scene to an animatable one. Surface mesh extraction (from volume density), tetrahedralisation and simulation are automated using off-the-shelf software, but manual animation is equally possible.
• Figure 4: Our volumetric face model extends the blendshapes of a traditional face 3DMM model from a surface (blue mesh) to a tetrahedral partition of the volume around it (green). The tetrahedral volume defines the support of the deformation and can be extended to cover hair, headphones or headgear (red).
• Figure 5: The two planes delineating rigid mouth regions. Tetrahedral mesh not shown for clarity, reference teeth geometry for illustration only.