Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deep Non-rigid Structure-from-Motion: A Sequence-to-Sequence Translation Perspective

Hui Deng, Tong Zhang, Yuchao Dai, Jiawei Shi, Yiran Zhong, Hongdong Li

TL;DR

This work reframes non-rigid structure-from-motion as a sequence-to-sequence translation problem, reconstructing a full 3D non-rigid sequence from a 2D sequence. It introduces a two-module architecture: a Shape/Motion Predictor for coarse per-frame estimates and a Context Layer with multi-head attention to enforce self-expressiveness and temporal structure across the entire sequence, enabling end-to-end self-supervised training. Temporal encoding (absolute and relative) and a nuclear-norm-based低-rank regularizer on the reshuffled shape matrix $\\boldsymbol{S}^{\sharp}$ together with a canonicalization loss and a smoothness term yield robust sequence-level reconstruction. Across CMU Mocap, Human3.6M, and InterHand2.6M, the method achieves state-of-the-art accuracy and demonstrates length-scalable inference, highlighting the practical impact for efficient, sequence-aware NRSfM in diverse domains.

Abstract

Directly regressing the non-rigid shape and camera pose from the individual 2D frame is ill-suited to the Non-Rigid Structure-from-Motion (NRSfM) problem. This frame-by-frame 3D reconstruction pipeline overlooks the inherent spatial-temporal nature of NRSfM, i.e., reconstructing the whole 3D sequence from the input 2D sequence. In this paper, we propose to model deep NRSfM from a sequence-to-sequence translation perspective, where the input 2D frame sequence is taken as a whole to reconstruct the deforming 3D non-rigid shape sequence. First, we apply a shape-motion predictor to estimate the initial non-rigid shape and camera motion from a single frame. Then we propose a context modeling module to model camera motions and complex non-rigid shapes. To tackle the difficulty in enforcing the global structure constraint within the deep framework, we propose to impose the union-of-subspace structure by replacing the self-expressiveness layer with multi-head attention and delayed regularizers, which enables end-to-end batch-wise training. Experimental results across different datasets such as Human3.6M, CMU Mocap and InterHand prove the superiority of our framework.

Deep Non-rigid Structure-from-Motion: A Sequence-to-Sequence Translation Perspective

TL;DR

This work reframes non-rigid structure-from-motion as a sequence-to-sequence translation problem, reconstructing a full 3D non-rigid sequence from a 2D sequence. It introduces a two-module architecture: a Shape/Motion Predictor for coarse per-frame estimates and a Context Layer with multi-head attention to enforce self-expressiveness and temporal structure across the entire sequence, enabling end-to-end self-supervised training. Temporal encoding (absolute and relative) and a nuclear-norm-based低-rank regularizer on the reshuffled shape matrix together with a canonicalization loss and a smoothness term yield robust sequence-level reconstruction. Across CMU Mocap, Human3.6M, and InterHand2.6M, the method achieves state-of-the-art accuracy and demonstrates length-scalable inference, highlighting the practical impact for efficient, sequence-aware NRSfM in diverse domains.

Abstract

Directly regressing the non-rigid shape and camera pose from the individual 2D frame is ill-suited to the Non-Rigid Structure-from-Motion (NRSfM) problem. This frame-by-frame 3D reconstruction pipeline overlooks the inherent spatial-temporal nature of NRSfM, i.e., reconstructing the whole 3D sequence from the input 2D sequence. In this paper, we propose to model deep NRSfM from a sequence-to-sequence translation perspective, where the input 2D frame sequence is taken as a whole to reconstruct the deforming 3D non-rigid shape sequence. First, we apply a shape-motion predictor to estimate the initial non-rigid shape and camera motion from a single frame. Then we propose a context modeling module to model camera motions and complex non-rigid shapes. To tackle the difficulty in enforcing the global structure constraint within the deep framework, we propose to impose the union-of-subspace structure by replacing the self-expressiveness layer with multi-head attention and delayed regularizers, which enables end-to-end batch-wise training. Experimental results across different datasets such as Human3.6M, CMU Mocap and InterHand prove the superiority of our framework.
Paper Structure (30 sections, 1 theorem, 18 equations, 6 figures, 6 tables)

This paper contains 30 sections, 1 theorem, 18 equations, 6 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Traditional NRSfM Methods
  4. Deep NRSfM Methods
  5. Sequence Modeling
  6. Uniqueness of Our Contribution
  7. Method
  8. Problem Formulation and A Recap
  9. Framework Overview
  10. Shape and Motion Prediction
  11. Sequence Modeling and Self-expressiveness
  12. Modeling Complex Non-rigid Sequences.
  13. Modeling Temporal Encoding
  14. Shape Regularizer
  15. Experiments
  16. ...and 15 more sections

Key Result

Theorem 1

Suppose that a deformable shape $\mathbf{S}^\sharp \in \mathbb{R}^{F \times 3P}$ lies in a low-rank space, then we have: Here, $\mathbf{Q} \in \mathbb{R}^{3F\times 3F}$ is the per-frame rotation ambiguity matrix defined as $blkdiag(\mathbf{Q}_{1},\cdots,\mathbf{Q}_{F}), \mathbf{Q}_{i} \in SO(3)$. The equal sign holds if and only if $\mathbf{Q}$ contains only one rotation component $\mathbf{Q}_1=\

Figures (6)

  • Figure 1: A conceptual illustration of our sequence-to-sequence NRSfM reconstruction framework, where we take the 2D frame sequence as a whole to predict the deforming 3D shape sequence.
  • Figure 2: An overview of our proposed deep sequence-to-sequence NRSfM framework. Our framework consists of two core modules: Shape/Motion predictor (b) for estimating the initial 3D shape and camera motion from a single frame, and Context Layer (c) for adjusting the 3D shape sequence by exploiting the inherent structure within the whole 3D sequence.
  • Figure 3: Qualitative comparison between our method and frame-to-shape method C3dpo novotny2019c3dpo and DNRSfM kong2020deep on Human3.6M, where we use the red line to mark the error between the predicted 3D shape and the ground truth 3D shape.
  • Figure 4: Visualization of several methods on Human3.6M with detected 2D keypoint and Interhand2.6M dataset. Our method is effective in reconstructing different kinds of non-rigid objects.
  • Figure 5: Performance of each method with the increase in the length of the inference sequence for a certain training length. The stability of the Base+nopos scheme in terms of length scalability and the optimal performance of the Base+abs+smooth scheme compared to other setups can be more intuitively seen in this figure.
  • ...and 1 more figures

Theorems & Definitions (1)

  • Theorem 1