Spectral Meets Spatial: Harmonising 3D Shape Matching and Interpolation

TL;DR

A unified framework to predict both point-wise correspondences and shape interpolation between 3D shapes is presented, which outperforms previous state-of-the-art methods for both shape matching and interpolation, even compared to supervised approaches.

Abstract

Although 3D shape matching and interpolation are highly interrelated, they are often studied separately and applied sequentially to relate different 3D shapes, thus resulting in sub-optimal performance. In this work we present a unified framework to predict both point-wise correspondences and shape interpolation between 3D shapes. To this end, we combine the deep functional map framework with classical surface deformation models to map shapes in both spectral and spatial domains. On the one hand, by incorporating spatial maps, our method obtains more accurate and smooth point-wise correspondences compared to previous functional map methods for shape matching. On the other hand, by introducing spectral maps, our method gets rid of commonly used but computationally expensive geodesic distance constraints that are only valid for near-isometric shape deformations. Furthermore, we propose a novel test-time adaptation scheme to capture both pose-dominant and shape-dominant deformations. Using different challenging datasets, we demonstrate that our method outperforms previous state-of-the-art methods for both shape matching and interpolation, even compared to supervised approaches.

Figures (20)

• Figure 1: Top: 3D shape interpolation. Compared to the SOTA shape interpolation method NeuroMorph eisenberger2021neuromorph, our method obtains more reliable interpolation even under large non-isometry. Bottom left: 3D shape matching. The point-wise correspondences found by the SOTA shape matching method URSSM cao2023unsupervised contains local mismatches. In contrast, our method enables smooth shape matching. Bottom right: 3D shape matching and interpolation. Our method is the first unsupervised method that obtains both accurate correspondences (shown as texture transfer) and realistic interpolation that capture both the pose-dominant and shape-dominant deformations.
• Figure 2: Method overview. First, the Siamese feature extractor $\mathcal{F}_{\theta}$ is used to extract features $\mathbf{F}_{\mathcal{X}}, \mathbf{F}_{\mathcal{Y}}$ from input shapes $\mathcal{X}, \mathcal{Y}$, respectively. The extracted features are then used to compute both bidirectional functional maps $\mathbf{C}_{\mathcal{XY}}, \mathbf{C}_{\mathcal{YX}}$ and point-wise maps $\mathbf{\Pi}_{\mathcal{XY}}, \mathbf{\Pi}_{\mathcal{YX}}$. Afterwards, the computed point-wise maps are used to bring the shape into correspondences. In the context of shape interpolation, a series of time steps $t$ is sampled and fed into the interpolator $\Omega_{\gamma}$ together with shape information to predict a series of interpolated shapes ${X}(t), Y(t)$. During training the spectral loss $L_{\mathrm{spectral}}$ is used to regularise the predicted functional maps and point maps, while the spatial loss $L_{\mathrm{spatial}}$ is used to regularise the interpolation trajectories of both shapes.
• Figure 3: Visualisation of our test-time adaptation. Starting from pose-dominant deformation based on ARAP, we obtain $\mathbf{X}(t, t_s=0)$ (i.e. the last row). Afterwards, we optimise the shape-dominant deformation field $\mathbf{\Delta}_{s}(t)$ for each sampled time point. Finally, the linear interpolation is performed to obtain the shape-dominant interpolation trajectories (shown by arrows).
• Figure 4: Near-isometric shape matching on FAUST, SCAPE. Proportion of correct keypoints (PCK) curves and corresponding area under curve (scores in the legend) of our method in comparison to existing state-of-the-art methods.
• Figure 5: Non-isometric matching on SMAL, DT4D-H inter-class datasets. Our method sets to new state of the art by a large margin based on the combination of spectral and spatial maps.