Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Non-Rigid Structure-from-Motion via Differential Geometry with Recoverable Conformal Scale

Yongbo Chen, Yanhao Zhang, Shaifali Parashar, Liang Zhao, Shoudong Huang

TL;DR

This work tackles non-rigid structure-from-motion under conformal deformations by grounding the reconstruction in differential geometry. It introduces Con-NRSfM, a framework that decouples depth and local conformal scale through rotationally invariant connections and metric-preserving moves, solved with a parallel, separable optimization over a well-connected image-warp graph. A point-wise NLLS formulation yields 21 virtual measurements per edge, and a self-supervised encoder–decoder reconstructs dense textured point clouds, boosting both accuracy and practicality in deformable-SLAM contexts. Extensive simulations and real-data experiments demonstrate improved reconstruction quality and robustness across conformal and non-isometric deformations, while maintaining reasonable computation times. The approach advances deformable 3D reconstruction by synergizing differential geometry with learning-based densification, offering a new path for online visual deformable SLAM and dense texture-rich models.

Abstract

Non-rigid structure-from-motion (NRSfM), a promising technique for addressing the mapping challenges in monocular visual deformable simultaneous localization and mapping (SLAM), has attracted growing attention. We introduce a novel method, called Con-NRSfM, for NRSfM under conformal deformations, encompassing isometric deformations as a subset. Our approach performs point-wise reconstruction using 2D selected image warps optimized through a graph-based framework. Unlike existing methods that rely on strict assumptions, such as locally planar surfaces or locally linear deformations, and fail to recover the conformal scale, our method eliminates these constraints and accurately computes the local conformal scale. Additionally, our framework decouples constraints on depth and conformal scale, which are inseparable in other approaches, enabling more precise depth estimation. To address the sensitivity of the formulated problem, we employ a parallel separable iterative optimization strategy. Furthermore, a self-supervised learning framework, utilizing an encoder-decoder network, is incorporated to generate dense 3D point clouds with texture. Simulation and experimental results using both synthetic and real datasets demonstrate that our method surpasses existing approaches in terms of reconstruction accuracy and robustness. The code for the proposed method will be made publicly available on the project website: https://sites.google.com/view/con-nrsfm.

Non-Rigid Structure-from-Motion via Differential Geometry with Recoverable Conformal Scale

TL;DR

This work tackles non-rigid structure-from-motion under conformal deformations by grounding the reconstruction in differential geometry. It introduces Con-NRSfM, a framework that decouples depth and local conformal scale through rotationally invariant connections and metric-preserving moves, solved with a parallel, separable optimization over a well-connected image-warp graph. A point-wise NLLS formulation yields 21 virtual measurements per edge, and a self-supervised encoder–decoder reconstructs dense textured point clouds, boosting both accuracy and practicality in deformable-SLAM contexts. Extensive simulations and real-data experiments demonstrate improved reconstruction quality and robustness across conformal and non-isometric deformations, while maintaining reasonable computation times. The approach advances deformable 3D reconstruction by synergizing differential geometry with learning-based densification, offering a new path for online visual deformable SLAM and dense texture-rich models.

Abstract

Non-rigid structure-from-motion (NRSfM), a promising technique for addressing the mapping challenges in monocular visual deformable simultaneous localization and mapping (SLAM), has attracted growing attention. We introduce a novel method, called Con-NRSfM, for NRSfM under conformal deformations, encompassing isometric deformations as a subset. Our approach performs point-wise reconstruction using 2D selected image warps optimized through a graph-based framework. Unlike existing methods that rely on strict assumptions, such as locally planar surfaces or locally linear deformations, and fail to recover the conformal scale, our method eliminates these constraints and accurately computes the local conformal scale. Additionally, our framework decouples constraints on depth and conformal scale, which are inseparable in other approaches, enabling more precise depth estimation. To address the sensitivity of the formulated problem, we employ a parallel separable iterative optimization strategy. Furthermore, a self-supervised learning framework, utilizing an encoder-decoder network, is incorporated to generate dense 3D point clouds with texture. Simulation and experimental results using both synthetic and real datasets demonstrate that our method surpasses existing approaches in terms of reconstruction accuracy and robustness. The code for the proposed method will be made publicly available on the project website: https://sites.google.com/view/con-nrsfm.

Paper Structure

This paper contains 37 sections, 3 theorems, 42 equations, 27 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. NRSfM methods
  4. Related applications
  5. Mathematical Modeling Preliminary
  6. Notations
  7. Perspective Projection and Image Embedding
  8. Deformation Mapping
  9. Selected Image Warp
  10. Moving Frames and Connections
  11. Metric Preservation under Conformal Deformation
  12. Connections under Conformal Deformation
  13. Con-NRSfM Algorithm
  14. Point-wise Solution
  15. Separable Framework using Parallel Strategy
  16. ...and 22 more sections

Key Result

Theorem 1

For a conformal deformation, $\mathbf{J}_{\Psi_{21}}=\lambda \mathbf{R}$, the relation between connections ${\Gamma}(\Phi_1)$ and $\Gamma(\Phi_2\circ\eta_{12})$ is invariant to rotation $\mathbf{R}$ and is not invariant to $\lambda$, given by

Figures (27)

  • Figure 1: A 2-view model for NRSfM.
  • Figure 2: Point-wise solution using graph optimization.
  • Figure 3: Parallel method to solve the point-based problems.
  • Figure 4: Parallel method to solve the depth recovery.
  • Figure 5: Self-supervised network for depth recovery.
  • ...and 22 more figures

Theorems & Definitions (11)

  • Theorem 1
  • proof
  • Corollary 1
  • proof
  • Corollary 2
  • proof
  • Remark 1
  • proof
  • proof
  • proof
  • ...and 1 more