Deformable NeRF using Recursively Subdivided Tetrahedra

TL;DR

DeformRF addresses the lack of explicit deformability in NeRF representations by integrating a manipulable tetrahedral mesh with high-quality feature-grid rendering. It introduces a two-stage training workflow and the concept of recursively subdivided tetrahedra to enable multi-resolution encoding while storing only a coarse initial mesh. A hierarchical barycentric inference scheme allows efficient interpolation across levels without storing dense subdivided meshes. Empirical results on synthetic and real data demonstrate strong novel view synthesis and robust deformation/rigged-animation capabilities, with favorable memory characteristics compared to prior tetrahedral approaches. Overall, the approach extends NeRFs to explicit object manipulation while preserving photorealistic rendering quality.

Abstract

While neural radiance fields (NeRF) have shown promise in novel view synthesis, their implicit representation limits explicit control over object manipulation. Existing research has proposed the integration of explicit geometric proxies to enable deformation. However, these methods face two primary challenges: firstly, the time-consuming and computationally demanding tetrahedralization process; and secondly, handling complex or thin structures often leads to either excessive, storage-intensive tetrahedral meshes or poor-quality ones that impair deformation capabilities. To address these challenges, we propose DeformRF, a method that seamlessly integrates the manipulability of tetrahedral meshes with the high-quality rendering capabilities of feature grid representations. To avoid ill-shaped tetrahedra and tetrahedralization for each object, we propose a two-stage training strategy. Starting with an almost-regular tetrahedral grid, our model initially retains key tetrahedra surrounding the object and subsequently refines object details using finer-granularity mesh in the second stage. We also present the concept of recursively subdivided tetrahedra to create higher-resolution meshes implicitly. This enables multi-resolution encoding while only necessitating the storage of the coarse tetrahedral mesh generated in the first training stage. We conduct a comprehensive evaluation of our DeformRF on both synthetic and real-captured datasets. Both quantitative and qualitative results demonstrate the effectiveness of our method for novel view synthesis and deformation tasks. Project page: https://ustc3dv.github.io/DeformRF/

Figures (8)

• Figure 1: Overview of DeformRF. (a) Given that a ray intersects with a tetrahedral mesh, we proceed with ray marching while retaining the sample points within the tetrahedron. (b) For each sample, we perform barycentric interpolation at each level and combine the feature vectors from all levels to create a complete feature vector. In this process, the computation of the barycentric coordinates is conducted iteratively. (c) In the two-stage training process, we first acquire a coarse mesh and then enhance training through increased subdivisions. (d) Our method support physically-based simulation and rigged animation.
• Figure 2: Subdivision of a Tetrahedron. Given a tetrahedron $T$, we subdivide it into eight smaller tetrahedra $T_k$, for $k \in \{0, \ldots, 7\}$, by connecting the midpoints of each edge.
• Figure 3: Qualitative Comparisons on the Synthetic NeRF Dataset NeRF.
• Figure 4: Qualitative Comparisons on the Tanks and Temples Dataset tnt.
• Figure 5: Predicted Accumulation Map Comparisons on the Synthetic NeRF Dataset NeRF and the Tanks and Temples Dataset tnt.