4Deform: Neural Surface Deformation for Robust Shape Interpolation

TL;DR

4Deform addresses the problem of robust shape interpolation between sparse, potentially topology-changing point clouds by learning a time-varying implicit surface $\phi(\mathbf{x},t)$ and an accompanying velocity field $\mathcal{V}(\mathbf{x},t)$. It integrates a correspondence module based on unsupervised shape matching with a Modified Level-Set framework, enforcing physics-inspired losses (Distortion and Stretching) to produce physically plausible intermediate shapes without requiring ground-truth intermediates. The method demonstrates state-of-the-art performance on isometric, non-isometric, and partial data, and enables practical applications such as 4D sequence upsampling of real Kinect data and high-resolution mesh deformation, even for real-world noisy inputs. The contributions include (i) a data-driven framework relying on imprecise correspondences, (ii) two novel losses enforcing physical plausibility in implicit deformation, and (iii) demonstrated generalization to real-world sequences and extrapolation, with publicly available code forthcoming.

Abstract

Generating realistic intermediate shapes between non-rigidly deformed shapes is a challenging task in computer vision, especially with unstructured data (e.g., point clouds) where temporal consistency across frames is lacking, and topologies are changing. Most interpolation methods are designed for structured data (i.e., meshes) and do not apply to real-world point clouds. In contrast, our approach, 4Deform, leverages neural implicit representation (NIR) to enable free topology changing shape deformation. Unlike previous mesh-based methods that learn vertex-based deformation fields, our method learns a continuous velocity field in Euclidean space. Thus, it is suitable for less structured data such as point clouds. Additionally, our method does not require intermediate-shape supervision during training; instead, we incorporate physical and geometrical constraints to regularize the velocity field. We reconstruct intermediate surfaces using a modified level-set equation, directly linking our NIR with the velocity field. Experiments show that our method significantly outperforms previous NIR approaches across various scenarios (e.g., noisy, partial, topology-changing, non-isometric shapes) and, for the first time, enables new applications like 4D Kinect sequence upsampling and real-world high-resolution mesh deformation.

Figures (12)

• Figure 1: 4Deform takes a sparse temporal sequence of point clouds as input and generates realistic intermediate shapes. Starting from just pairs of point clouds and estimated sparse, noisy correspondences (indicated using colors in the point clouds), our method obtains realistic long-range interpolations, even for shapes with changing topology (e.g., the human-object interaction in the top row), and can generalize the interpolation results to real-world data (Kinect point clouds in the second row). Meanwhile, our method can handle non-isometrically deformed shapes (bottom left) as well as partial shapes (bottom right).
• Figure 2: Pipeline of 4Deform: Given a temporal sequence of inputs, we initialize a latent vector to each point cloud. Then the network takes pairs of point clouds $\mathcal{P}_0$ and $\mathcal{P}_1$ (with sparse correspondences), together with the concatenated latent vector $\mathbf{z}_0$ and $\mathbf{z}_1$ as input. At training time, we jointly optimize two neural fields: a time-varying implicit representation (Implicit Net $\phi$) and a velocity field (Velocity Net $\mathcal{V}$) with proposed geometric and physical constraints losses. Conditioning on a time stamp $t$, we instantaneously obtain a continuous time-varying signed distance function (SDF), an offset of the input toward the target (velocity field).
• Figure 3: Large Deformations. 4Ddeform handles large deformations better than previous works, providing one order of magnitude less area distortion, even compared to mesh-based ones (LIMP Cosmo2020). In the top row, we visualize the error in the estimated input correspondence.
• Figure 4: Non-isometric deformation. We deform two different animals from SMAL, relying on a noisy correspondence (top row). Compared to the previous methods, our method results in plausible deformations, while preserving thin geometric details (e.g., legs).
• Figure 5: Partial shape deformation. We consider the case in which one of the input shapes is only partially available while having noisy correspondences (correspondence error visualized in the top row). Other methods often collapse the unseen part or create unreasonable stretches. Similar effects are observed when we remove some of our novel losses. Our method provides plausible interpolations, both for the visible and missing parts.