4DRecons: 4D Neural Implicit Deformable Objects Reconstruction from a single RGB-D Camera with Geometrical and Topological Regularizations

TL;DR

4DRecons introduces a 4D neural implicit representation for textured deformable objects reconstructed from a single RGB-D sequence. It jointly optimizes a data term with two regularizers: an ARAP-based deformation term that propagates observed geometry via correspondences between adjacent textured implicit surfaces, and a topology regularization based on persistent diagrams to enforce consistent topology over time. The method employs a four-stage optimization that initializes from partial observations, then alternates geometry and color refinement under regularization, ultimately delivering a complete, self-intersection-free 4D surface. Experiments across multiple datasets show substantial improvements in both geometry and texture quality over state-of-the-art baselines, including cases with large deformations and topology changes.

Abstract

This paper presents a novel approach 4DRecons that takes a single camera RGB-D sequence of a dynamic subject as input and outputs a complete textured deforming 3D model over time. 4DRecons encodes the output as a 4D neural implicit surface and presents an optimization procedure that combines a data term and two regularization terms. The data term fits the 4D implicit surface to the input partial observations. We address fundamental challenges in fitting a complete implicit surface to partial observations. The first regularization term enforces that the deformation among adjacent frames is as rigid as possible (ARAP). To this end, we introduce a novel approach to compute correspondences between adjacent textured implicit surfaces, which are used to define the ARAP regularization term. The second regularization term enforces that the topology of the underlying object remains fixed over time. This regularization is critical for avoiding self-intersections that are typical in implicit-based reconstructions. We have evaluated the performance of 4DRecons on a variety of datasets. Experimental results show that 4DRecons can handle large deformations and complex inter-part interactions and outperform state-of-the-art approaches considerably.

Figures (13)

• Figure 1: Pipeline overview of 4DRecons, which performs a four-stage optimization procedure. The first stage initializes the geometry field by fitting the input data. The second stage enforces deformation and topology regularizations to improve the geometry field. The third stage initializes the color field while fixing the geometry field. The last stage jointly refines the geometry field and the color field.
• Figure 2: The filtration processes of two SDF volumes, resulting in two comparable PD plots that we enforce alignment. Our method allows the alignment gradients to back-propagate and directly manipulate the underlining SDF volumes to synchronize their topological signatures.
• Figure 3: (a) We first discretize an implicit surface in frame $t$ into a triangular mesh. Then we solve an optimization problem to determine the correspondences of the vertices of this mesh in the next frame $t+1$ and the previous frame $t-1$. These correspondences are then used to define the deformation regularization term. (b) Comparisons of the geometry field reconstruction with (right) and without (left) the deformation regularization term $l_{\textup{def}}(\theta)$ defined in \ref{['Subsec:Def:Reg:Term']}.
• Figure 4: Qualitative comparisons of geometry field reconstruction on sequences without topological changes among (a) DynamicFusion DBLP:conf/cvpr/NewcombeFS15, (b) NDR Cai2022NDR and (c) 4DRecons.
• Figure 5: Qualitative evaluation on sequences with topology changes.